“I am feeling blue… and red”: Exploring alternative ways of feeling colours through a wearable haptic artefact

Abstract

Wearable haptic artefacts are more and more present in our everyday life. They increasingly mediate our perception, transforming the ways in which we interact with ourselves, the environment and others. Wearable haptic artefacts can sense the body and its surroundings, impacting our subjectivity by shaping perceptions, actions, and behaviours. Such ability to influence our senses is helpful in creating engaging multisensory experiences. In my work, I explore the technological mediation between the human body and its surroundings by facilitating new relationships among the senses. This article presents a wearable haptic artefact in the shape of a glove called “I am feeling blue… and red”, which was designed to enable users to experience colour through touch. The goal is to suggest a mode of colour experience and break preconceptions one might have towards the relationship there is between touch and colours. Through an autoethnographic account, the article reflects on the artefact’s design process and user interaction. It proposes strategies for developing unique multisensory experiences using wearable haptic artefacts. These strategies can assist artists and designers who want to create captivating interactions by connecting the senses of sight and touch.

Introduction

Advancements in wearable technology have expanded human perceptual possibilities, broadening the range of human experiences and providing unique opportunities for exploration and discovery. In addition, wearable technologies have brought about a paradigm shift in how we interact with our surroundings, targeting the entire human body as a site for interaction. Wearable technology is characterised by its proximity to the human body and its technical capability to sense the user and their surroundings, thus mediating interaction with the world (Verbeek, 2021).

Those characteristics, namely the proximity to the body and the ability to sense the body and its surroundings, have significant implications for our subjectivity. This is because, during an interaction, the user and the wearable technology influence one another, with the artefact responding to the user's actions and the user responding to the artefact’s feedback. The artefact affects the body, which could potentially evoke sensations and emotions in the wearer. Next, I briefly discuss wearable haptic artefacts (WHAs) and their applications. Following that, through an autoethnographic account, I reflect on an artefact designed to enable the wearer to perceive colours through the sense of touch and explore the ways one might use them to interact with the world and through engaging in multisensory experiences.

Wearable haptic artefacts and their applications

Haptic artefacts use tactile or force-feedback technology to stimulate a person's sense of touch by applying stimuli to their body. When such technology is designed to be worn in and around the living human body, it is referred to as wearable haptic artefacts (Kortum, 2008). This technology can be used for various purposes, including in medical applications, physical rehabilitation and in immersive virtual environments for simulating bodily sensations. However, I approach wearable haptics from a less functional and more poetic perspective in my artistic practice. I focus on creating unusual or unfamiliar sensations coupled with uncommon gestures to liberate touch from a utilitarian purpose.

Additionally, this strategy directs the wearer’s attention to a sensation, material, context, or phenomenon that might be foreign to them. The goal is to break the take-for-granted attitude one might have towards their body and surroundings. The work examined in this article uses an artefact to mediate the wearer’s relationship with colours, physiologically perceived through sight, by exciting another sensory modality: touch.

The relationship between colour and vibrotactile stimuli has been explored across various contexts. An example is an exploration of systems and artefacts that uses tactile patterns to represent colours, allowing visually impaired people to experience colour. Past research by Shin et al., (2020) devised distinct 3D-printed textured patterns to represent distinct colours. Their objective was to establish what are the texture-colour associations that could be actively sensed through fingertip movements, akin to the act of reading braille. Delazio et al. (2017), conducted a meticulously designed experiment of multimodal sensory experiences to investigate the connection between vibrotactile stimuli and colour perception. Their findings indicated that low-frequency vibrations elicited violet hues, while high-frequency vibrations evoked green hues. In both studies, the research focus was on hands, emphasising functionality and striving to establish a reliable translation system between sensory modalities. While the present work also employs an arbitrary correlation between colour and vibration, it does not aim to discern various colours for the purpose of acquiring additional information.

Inspired by sensory substitution and augmentation systems (Shull and Damian, 2015), the work presented in this paper draws from the notion of introducing new sensory inputs to the somatosensory system. While such systems are commonly employed in rehabilitation contexts to address sensory loss, the emphasis here lies in using WHAs to explore an enhanced sensorial experience by enabling the user to engage in a poetic and playful relationship with their surroundings.

The design of “I am feeling blue… and red.”

“I am feeling blue… and red” is a flexible fabric glove that has built-in electronic components. The design of the glove permits it to conform with movement, allowing active and explorative hand gestures. To ensure that flexibility is not compromised in the design, I opted to handcraft an electronic circuit using embroidery techniques with a conductive thread. The glove also comprises three coin vibration motors and one colour sensor, with a Flora-wearable electronic platform orchestrating the interaction (Industries, 2023). The sensor detects the colour, and the microcontrolled board translates the data into the RGB colour code. These data are then mapped to the coin vibration motors that vibrate in intensities that depend on the amount of Red, Green, or Blue detected. The location of the electronic components is illustrated in Figure 1. The colour sensor is located on the fingertip, the microcontroller board and battery are located on the forearm, and the vibration motors are located on the inner side of the forearm.

Figure 1.

“I am feeling blue… and red” WHA prototype: components (1- colour sensor TCS34725, 2- Microcontrolled board Flora, 3- Coin vibrators, 4- battery).

The glove is made of denim, and its design introduces deliberate modifications to the finger arrangement, resulting in a unique tactile experience. The design deliberately disrupts conventional exploratory hand gestures. Specifically, some fingers are immobilised, and some are exposed (see Figures 1 and 2). This configuration enables the wearer to perceive the nuanced textural qualities of the object through the bare fingers, however, at the expense of limited gestural dexterity and tactility imposed by the immobilised fingers.

Figure 2.

Interaction between the glove and the object (notebook cover), mediated by the WHA.

User interaction analysis

The glove proposes a novel relationship between sight and touch, freeing touch from utilitarian purposes. I hypothesise that using technology to enable colour perception through touch creates a space for creativity and reflection by focusing on the relationships between human and their surroundings the new technologically augmented senses afford. To better understand how the glove enables these new relationships, I will describe two distinct interactions facilitated by the glove: (1) an interaction between the glove and an object and (2) an interaction between the glove and a human body. This provides insights useful for designing human-environment multisensory experiences. Although the glove was initially designed to aid the artistic activity (i.e., as an artefact that facilitated a participatory activity to inspire poetic interpretation of colour), the artefact and its characteristics can be adapted and applied in research and experience design.

Interaction between the glove and an object

Interaction narrative: It is a pleasant afternoon as I slip on the glove. I am alone, and lounge music plays in the background. On the table is a notebook covered with yellow fur, featuring a lion made of different fabric textures and embroidering on the cover. After turning on the electronic circuit, the white light on the tip of the indicator finger catches my attention. My arm vibrates as I approach the notebook, and my bare thumb explores the texture while the indicator illuminates the surface. The vibration intensity on my arm varies as my fingers move along the surface, accompanied by different sound variations from the vibration motors. I explored the notebook for a few minutes before putting it back on the table. The vibration stops.

Reflections: The vibration sensations conveyed an extension of the notebook cover’s texture. The colour difference was translated through the vibration, enabling me to distinguish the yellows I touched without looking at the object. The alteration in vibration in relation to colours prompts further exploration of the object, facilitating a better comprehension of its material properties and potential affordances. This feature is valuable in multimodal experiences that deal with colours and touch representations because it encourages users to interact with the object in a more exploratory manner. By stimulating curiosity and motivating users to engage in probing gestures, the vibration feedback in relation to colour differences enhances overall sensory engagement

Figure 3.

Interaction between the glove and the human body, mediated by the WHA.

.

Interaction between the glove and the human body

Interaction narrative: In the same environment as the previous experience, I wear the glove and begin touching my other arm as soon as the light on the fingertip illuminates. To my surprise, the glove reacts to my skin colour. I ponder the ethical implications of this behaviour and wonder if it would detect other skin tones. Next, I touch my colourful tattoos, feeling different vibration sensations as my finger approaches them Figure 3. I also touch my clothes and continue exploring my arms and clothes for about 10 minutes.

Reflections: In physiology literature, touch is defined as the feeling experienced when the skin is exposed to mechanical, thermal, chemical, or electrical stimuli (Cholewiak and Collins, 1991). For this to happen, physical contact with the skin is required. However, when mediated by the glove, at times, the sensor failed to detect the colour of the skin when the finger touched a surface, which improved with finger movement, especially when it was slightly elevated from the surface, suggesting a mode of touching without direct contact with the material. Engaging with my skin’s tactile vibrations and exploring the intricate interplay of tones and textures imprinted within it provides a contemplative space for reflecting upon its distinct characteristics and vibrant colours, fostering a deepened understanding and appreciation of the intricate interplay between the physicality of the skin and the aesthetic qualities it embodies.

This experience can be utilised in a social setting with two or more gloves used by a group of people to explore body diversity. I will develop this idea further in the next section.

Future directions and limitations

“I am feeling blue… and red” used in social interactions can foster the exploration of the singularities inherent in each individual’s skin and body. Through vibrations, one gains the ability to discern and interpret the diverse marks, moles, scars, tattoos, and tonal variations present on others, thereby enabling fresh insights and perspectives. This unique mode of engagement holds the potential to cultivate new emotional connections and foster a heightened appreciation for the intricacies that make each person unique. For example, empathy, understanding and unity in the context of the human body, fashion styles and identity. Nonetheless, when setting up such experiment, it is important to consider that this exploration should always be consensual. Facilitating such engagement, which is vital in any multisensory experience, can be achieved through the playful exploration of different colours via touch. Furthermore, this enhanced aesthetic appreciation of colours can serve as a design element for social interactions, offering a rich material for creative and meaningful engagements.

The interaction with skin elicited some ethical considerations. The sensor was not very accurate in detecting darker tones in my tattoos, which made me wonder how well it would perform in detecting darker skin tones. This flaw highlighted how technologies must be developed to fully embrace diversity and inclusion. One might argue that detecting skin tones might not have been the primary goal of the sensor. However, technologies can serve purposes other than those envisioned by the developers, and their ethical implications should be accounted for to avoid creating experiences that reinforce discrimination.

The current glove has limited tactile resolution, prompting the question of what new sensations a higher tactile resolution might bring, that is, the use of more vibration motors. In addition, the ideas that arise from the ethical considerations explained above suggest future research on the effects of social interaction with more people mediated by the glove. Further research can investigate how the artefact can aid in the creation and promotion of social spaces and explore the potential social benefits. This requires a better understanding of the methodologies and strategies involved in utilising multisensory technologies to build and nurture environments that encourage dialogue, exchange and facilitate novel forms of communication and collaboration.

Conclusion

This article discusses the design of a wearable haptic device that converts colour differences into vibration patterns felt in the forearm and two interactions mediated by the device. The goal was to break the take-for-granted attitude one might have towards colour and to support creativity and reflection. Three interaction design strategies have been identified through an autoethnographic account of using the artefact. These strategies could help to create more engaging multisensory experiences. Firstly, Vibration feedback that is correlated with colour differences enhances overall sensory engagement by stimulating curiosity and motivating users to engage in probing gestures. Secondly, WHAs can enable a mode of touch that does not require direct physical contact. This type of interaction could be beneficial when direct touch is not desirable. Thirdly, using difference (colour) as a criterion for the interaction might foster empathy towards others, as noted in the reflection about different skin colours. These strategies are helpful to create multisensory experiences that provoke aesthetic appreciation, reflection, and discovery.