Sage Journals: Discover world-class research

Abstract

This paper explores the transformative potential of haptic design and human augmentation, inspired by Dr. Minamizawa's keynote address at the 2024 Emerging Media for Communicating Sustainable Development Goals conference (held in Yang Ming Chiao Tung University, 30 March, 2024). By integrating tactile feedback into prosthetics and wearable devices, and employing robotic avatars and virtual interfaces, these technologies bridge the physical and virtual realms, redefining presence and interaction. The convergence of haptic design with telexistence and human augmentation enhances sensory engagement and bodily awareness, challenging traditional boundaries between biology and technology. However, these advancements might also raise ethical considerations and societal concerns about privacy, accessibility, autonomy, and safety. The future of human augmentation demands responsible development and inclusive dialogue to ensure empowerment, equity, and collective progress. This paper highlights the need for a human-integration approach to navigate the opportunities and challenges presented by these emerging technologies.

Keywords

Haptic design embodied interaction telexistence human augmentation robotic avatar

Introduction

In the realm of technological advancement, the convergence of haptic design and human augmentation stands as a beacon of innovation, promising to revolutionize the way we interact with and perceive the world around us. Haptic design, with its emphasis on tactile feedback and sensory immersion, has long been recognized for its potential to enhance user experience in various domains, from entertainment and gaming to rehabilitation and healthcare. On the other hand, human augmentation, fueled by breakthroughs in fields such as robotics, prosthetics, and wearable technology, seeks to augment human capabilities, bridging the gap between humans and machines.

Dr. Kouta Minamizawa is a professor at Graduate School of Media Design in Keio University. He directs the KMD Embodied Media Project, where conduct research on technology, design and social deployment of haptics and embodied interaction to facilitate the transmission, augmentation, and creation of human bodily experiences through interdisciplinary endeavors. Dr. Minamizawa is widely recognized as a leading figure in the field of embodied interaction technologies. His scholarly contributions focus on advancing knowledge in haptic design and human augmentation, fostering interdisciplinary collaborations, and driving innovation in the realm of sensory enhancement. His expertise and leadership in this domain are instrumental in promoting academic discourse and research activities aimed at shaping the future of human–computer interaction and immersive technologies.

Below are a few thoughts stemming from Dr. Minamizawa's keynote address at the 2024 Emerging Media for Communicating Sustainable Development Goals (SDGs) conference (held in Yang Ming Chiao Tung University, 30 March, 2024).

Haptic design

In his presentation on “Haptic Designer / 触業”, Dr. Minamizawa elaborated on the multifaceted nature of tactile perception, emphasizing the array of stimuli received through fingertips and palms. These stimuli encompass sensations such as pressure, texture, temperature, and pain, collectively constituting the role of touch in sensory perception. Drawing an analogy to equipped sensors, he elucidated how haptic technology functions as a mechanical sensor, mirroring the sensory capabilities of human cells and nerves and translating them into digital data. Through network transmission and reproduction, this technology enables the sharing of tactile sensations across distant locations, akin to the transmission and playback of visual and auditory content. Dr. Minamizawa's presentation showcased successful outcomes from his research endeavors, including the TECHTILE Toolkit (Minamizawa et al., 2012) and the CNN-based Tactile Search Engine (Hanamitsu et al., 2015), underscoring haptic design's ability to bridge physical and virtual tactile experiences (Figure 1) . The evolution of haptic technologies, from rudimentary force feedback mechanisms to sophisticated tactile displays, has been propelled by advancements in engineering, material science, and neuroscience. While early research primarily focused on basic haptic feedback for simulating physical interactions in virtual environments, contemporary efforts delve into nuanced forms of tactile communication and sensory augmentation. Fundamental to comprehending haptic design are theoretical frameworks elucidating the perceptual and cognitive mechanisms underlying tactile sensation. Concepts such as active touch, pioneered by James J. Gibson (1962), emphasize the dynamic interplay between sensory input and motor action in haptic exploration. Psychophysical models like Weber's Law and the Just Noticeable Difference (JND) offer quantitative insights into tactile perception thresholds and discrimination abilities. Recent studies by Cheng et al. (2023) and Kappenman et al. (2021) have expanded upon these foundational theories, providing nuanced perspectives on the neural mechanisms of haptic perception and the integration of tactile feedback with other sensory modalities. These theoretical advancements deepen our understanding of haptic processing in the brain and inform the design and evaluation of haptic interfaces and applications.

Figure 1.

CNN (deep learning)-based tactile search engine (Hanamitsu et al., 2015).

Another exciting application of haptic technologies is their incorporation into societal settings, which can yield unexpected benefits and significantly advance embodied learning for children and deaf individuals. This innovation enriches their capacity to engage with previously inaccessible experiences, such as dance performances and musical displays. Dr. Minamizawa's related developments include the “Kinder Buru Buru Cushion,” a joint project with Toppan Printing Co., Ltd., and Froebel-kan Co., Ltd. (Shibasaki et al., 2020). This cushion vibrates in sync with the content of children's picture books (Figure 2). Another development, “Karada Tap,” transfers haptic sensations and rhythmic cues from the stage to audience seats, enabling deaf and disabled individuals to enjoy tap dance performances or stage plays (Shibasaki, et al., 2016) (Figure 3); and, FeelTech™ (Figure 4) (NTT Docomo, 2023), developed for NTT Docomo, digitizes touch and allows the sharing of tactile sensations with people in distant locations. As to how a camera captures video for display on a TV or how a microphone records audio for broadcasting through a radio, FeelTech™ facilitates real-time communication through touch, compensating for information that sight and hearing alone cannot fully convey (Figure 4) to enhance life and memory.

Figure 2.

Kinder Buru Buru Cushion (Minamizawa, 2018).

Figure 3.

Karada Tap (Li et al., 2024)

Figure 4.

Feeltech™ (NTT Docomo, 2023).

Moreover, the intersection of haptic design with the burgeoning field of human augmentation heralds a new era of sensory enhancement and bodily awareness. Human augmentation technologies, encompassing prosthetics, exoskeletons, and wearable devices, seek to augment or enhance human capabilities, bridging the gap between the biological and the technological. Haptic design plays a pivotal role in this convergence by providing tactile feedback that not only enriches user experience but also fosters a deeper sense of embodiment and body awareness. Research by Nishio et al. (2012) has demonstrated how haptic feedback can be integrated into prosthetic limbs to provide users with a more intuitive sense of proprioception and limb position, facilitating smoother motor control and enhancing overall body awareness. This symbiotic relationship between haptic design and human augmentation underscores the transformative potential of tactile feedback in fostering a more profound connection between humans and their augmented selves. As such, the exploration of haptic design in the context of human augmentation opens up new avenues for enhancing not only sensory perception but also telexistence and agency.

Telexistence

In Dr. Minamizawa's view, the concept of “Telexistence,” a groundbreaking fusion of “Tele” and “Existence,” redefines the boundaries of human interaction with remote environments. Through the utilization of robotic or virtual avatars, users navigate and engage with distant locales, transcending physical barriers and limitations. Highlighting his pioneering projects like the Telexistence Surveillance Vehicle in collaboration with Obayashi Corp. (Kamijo, et al., 2016) and the HUG Project in partnership with Ducklings Corp. (Saraiji, et al., 2015 ), the related technology not only revolutionizes disaster surveillance but also presents a solution to age-related challenges. By embodying human capabilities in alternative forms, telexistence opens new horizons in human-robotic or human-avatar synergy, promising to reshape our understanding of spatial relationships and redefine the very essence of presence in scenarios of agent-based interaction. The effectiveness of telexistence performance might highly depend on the user's sense of presence, wherein they feel immersed in the remote environment rather than their physical surroundings. This concept of presence is a subjective experience that requires focusing on a coherent set of stimuli provided by remote functions. Further elaboration is warranted regarding different levels of achieving the complex phenomenon of presence particularly from the embodiment perspective: the phenomenology includes the sense of self-location (SoS), the sense of agency (SoA), and the sense of body ownership (Braun et al., 2018; Kilteni et al., 2012).

Sense of self-location

In robotic teleoperation, the SoS plays a critical role in enhancing operator performance and task execution efficiency. Self-location, or the operator's perceived position relative to a remote environment, significantly influences the teleoperator's spatial awareness and interaction capabilities. These coordinate systems, such as head-centered, arm-centered, body-centered, and world-centered coordinates, switch unintentionally when humans perform different tasks (Andersen et al., 1993). Advanced teleoperation systems thus aim to create an immersive experience where the operator feels physically present at the robot's location. This immersive SoS can be achieved through high-fidelity sensory feedback, such as visual, auditory, and haptic cues. Effective self-location is essential for tasks requiring precise manipulation and navigation, as it directly impacts the operator's ability to make real-time decisions based on spatial relationships.

Several user interface factors that influence the SoS have been discussed in previous studies. One of the frequently examined factors is the field of view (FOV) (Nakano et al., 2021). The display perspective presented by the virtual or digital camera determines how users can remotely interact with the virtual / real environment, what they focus on, and how their controlled agents adapt to the context. The impact of first-person FOV versus third-person FOV on the SoS remains uncertain. Pfeiffer et al. (2013) suggest that developing full-body illusions (FBIs) from a third-person perspective (3PP-FBI) could better enhance the SoS. These 3PP-FBIs are characterized by self-identification with an entire body in extrapersonal space rather than with a body part in peripersonal space. Conversely, other researchers (Huang et al., 2017; Petkova et al., 2011) argue that collocation between the virtual and real body (first-person perspective) elicits a stronger SoS than non-collocated perspectives (third-person point of view). Haptic design is another potential approach to increasing the SoS. Synchronous tactile and visual stimuli have been shown to enhance the SoS with the remote agent (Salomon et al., 2013). The well-known Rubber Hand Illusion experiment (Botvinick & Cohen, 1998) uses such a conceptual model to demonstrate that self-location can be altered when synchronous visuo-proprioceptive correlations occur between a rubber hand (visual) and the hidden real hand (tactile). In the context of robotic teleoperation, recognizing the coordinate of the robotic hand can significantly improve operability.

Self-consciousness is another possible issue that has been discussed for its effects on the spatial relationship between the body consciousness and the agent (Lenggenhager et al., 2009; Serino et al., 2013). It encompasses self-identification and awareness of one's own body, influencing how individuals perceive their location and interact with their environment. This intrinsic awareness affects the ability to accurately judge spatial relationships and navigate effectively, particularly in contexts where real and virtual environments intersect, such as in robotic teleoperation and virtual reality. Most previous studies on artificial avatars have focused on using highly human-like avatars or mannequins to enhance the sense of presence and realism for the operator. While findings from these studies (including those on artificial limbs) are often referenced in robotic teleoperations (Kheddar, 2001; Kheddar et al., 2014), technological research in this field has generally concentrated more on the high fidelity of sensory feedback displays (such as tactile feedback) at the “master” station, task performance, and the motions of the robot in remote locations rather than on issues related to the operator's self-consciousness.

Sense of agency

SoA encapsulates the subjective sensation of instigating and regulating an action, epitomizing the sense of authorship evoked when stating phrases such as “I am the one steering this tank” or “I must have been the one to hit the ball.” As such, SoA, as delineated by Cornelio et al. (2022), involves the subjective feeling of initiating and controlling actions, allowing individuals to distinguish their own actions from those generated externally. While SoA is often associated solely with motor control, it also encompasses intentional aspects that extend beyond the body's confines. As indicated by Lopez-Sola et al. (2021), SoA is a belief-based action-effect paradigm suggesting that the user's intention to act and the resulting action are sufficient to generate a SoA, irrespective of how the action is executed, whether mentally or through physical motor movements. This broader understanding of SoA underpins various ethical and legal concepts, including moral responsibility, free will, and culpability.

In essence, the SoA emerges when individuals act as agents to take control of their own actions. Through our interactions with our bodies, we exhibit precise motor control and maintain awareness of our movements, exemplified by proprioception. Agency involves the coordination of bodily movements with our intentions and the perception of movement, intricately linked with awareness and planning (Blanke & Metzinger, 2009). This SoA extends to the manipulation of tools, where understanding sensorimotor control and the relationship between actions and outcomes is crucial. The alignment between intention and outcome further enhances this SoA. Even in scenarios involving the control of robots or virtual avatars, individuals retain a SoA despite deviations from physical reality (Kilteni et al., 2012). That is, this resilience is apparent even when interacting with non-realistic avatars or those substantially different from one's physical body. Although adjustments may be necessary when the effector (e.g. robot hands, avatar gestures) differs from the real body, individuals can adapt through processes like proprioceptive adjustments, motor learning, or recalibration of virtual space (Henriques & Cressman, 2012). However, ensuring a high level of perceptual-motor fidelity between individuals and their agents remains paramount.

Bodily ownership

In addition to the SoA achieved through regulating actions, body ownership is described as the feeling of owning one's body (or an artificial body) (Longo et al., 2008) and experiencing the body (real or artificial) as the source of sensations (Tsakiris, Prabhu, & Haggard 2006). Specifically, while SoA arises solely from voluntary actions, body ownership is experienced during both voluntary actions and passive experiences. Body ownership denotes the unique perceptual status of one's own body, imbuing bodily sensations with a distinctly personal quality in remote contexts and virtual reality, significantly impacting the sense of presence. Recently, the cognitive neuroscience community has increasingly agreed that the perception of one's own body in space critically relies on multisensory integration (Ehrsson, 2012; Kilteni et al., 2015; Lopez et al., 2008). That is, ownership illusions have demonstrated that users not only respond to threats to their virtual or rubber body parts (Botvinick & Cohen, 1998; Yuan & Steed, 2010), such as virtual arms (Slater et al,, 2008), legs (Kokkinara & Slater, 2014), or FBIs (Petkova et al., 2011), but also integrate information from afferents in the skin, muscles, and tendons, as well as visual, vestibular, and auditory signals, in the cortical convergence of body signals (Angelaki & Cullen, 2008; Hagura et al., 2007). These findings underscore the importance of bodily agency perception and the sensation of ownership.

While considerable attention has been devoted to examining the resemblance between the user and the controlled agent (Guterstam et al., 2015; Piryankova et al., 2014), there remains a notable gap in research regarding how structural changes in the agent's morphology impact users’ perceived body ownership. Besides visual and visuotactile stimuli, the design of avatar appearances significantly impacts the illusion of body transfer and the sense of body ownership. McCloud's (1993) Realism Triangle Model and Peirce's (1894) Theory of Signs are widely used frameworks for categorizing graphic interfaces and character appearances: By dividing representations into two dimensions: detailed features and abstraction level, McCloud (1993) divides visual representations based on their physical resemblance (Realism), causal relationships or metaphors (Index), and abstractions or arbitrary definitions (Symbol). According to Figure 5, the horizontal axis shows a decrease in the level of detailed features from left to right, with the most realistic images (such as photographs) located at the bottom left, presenting detailed features realistically. In contrast, the bottom right depicts iconic representations that utilize metaphors or references. Lastly, the geometric figures at the top represent abstract images, whose symbolic meanings are culturally or thematically influenced.

Figure 5.

Realism triangle model (McCloud, 1993).

Peirce's (1894) Theory of Signs classified symbols based on the principles of semiotics. Symbols are meaningful basic units defined in two parts: the signifier and the signified. The former is a sign with specific meaning (often represented as sounds or images) that evokes associations with specific objects or concepts, while the latter refers to the concept associated with this sign image. Although McCloud's Realism Triangle Model may differ in terminology, it is quite similar to Peirce's types of symbols in terms of representational meaning, namely realistic, metaphorical, and abstract symbols. Integrating the perspectives of these scholars, Rogers (1989) proposed four typologies that might more suitable for agent morphology design:

Resemblance: Direct similarity to the objects they represent, such as the lens icon for a phone camera.

Exampler: Common examples of the object category they represent, such as using a trowel or broom to represent gardening.

Symbolic: Conveying abstract concepts, such as a broken wine glass representing fragility.

Arbitrary: Having no inherent relationship with the object or concept, requiring learning of their relationship, such as the on/off button of a computer.

Above theoretical frameworks corroborate the critical impact of agent morphology on user experience in a virtual or a remote scenario. Agents closely resembling or personalized by the user can enhance the illusion of body transfer, body ownership, and presence (Gorisse et al., 2019; Waltemate et al., 2018). However, the findings of Lugrin et al. (2015) highlight the nuanced nature of this relationship. While robotic and cartoon agents excel in inducing a strong body transfer illusion, overly realistic humanoid agents may evoke discomfort due to the uncanny valley effect. This underscores the importance of understanding how different typologies of agent morphology influence user perception and experience in virtual environments, offering insights for optimizing design choices to create immersive and engaging embodiment experiences.

Human augmentation

Not a “Tool” but an “Integration”

Since the emergence of direct manipulation and graphical user interfaces / GUIs (Engelbart, 1968; Lipkie et al., 1982; Shneiderman, 1982), mainstream human–technology interaction has undergone gradual evolution. The foundational interaction model, GUIs have enabled more direct, efficient, and robust tool usage. Recent research in the field has fast towards pervasive and intelligent interaction, including agent-based interaction (Macy and Willer, 2002), embodied interaction (Dourish, 2001), and conversational interfaces (Kocaballi et al., 2019). Despite these advancements, GUIs maintain a clear delineation between the user and the system which view computing devices primarily as tools. Human augmentation presents a different paradigm shift that builds upon these established frameworks by prioritizing human action in interaction design. This approach entails enhancing human actions with technologies focused on perception and manipulation.

In his presentation, Dr. Minamizawa emphasizes the immense potential of human augmentation technology in digitally transforming human bodily experiences, encompassing sharing, enhancement, and creation. In addition to the telexistence experience, which enables control over action, sensation, and authorship, Dr. Minamizawa advocates for cybernetic avatars (CAs) not a conventional “tool,” but as “alternative bodies,” conceived through three core constructs: (1) cognitive augmentation, (2) parallel agency & experience sharing, and (3) collective abilities. Cognitive augmentation offers users an alternative body capable of dynamically harnessing their potential in response to varying situations and environments; Parallel agency & experience sharing present an alternative body equipped to perceive and engage across multiple spaces simultaneously. Lastly, collective abilities introduce an alternative body capable of amalgamating the diverse skills of multiple individuals, transcending individual limitations to achieve collective goals.

The “Moonshot” project stands as one of Dr. Minamizawa's most ambitious research endeavors, centered on these three constructs advancing CA technology to empower individuals in unlocking their full potential and facilitating the global exchange of diverse skills and experiences. Recognizing the complex social and ethical considerations inherent in the shared utilization of physical skills and experiences, the “Moonshot” project is dedicated to establishing the groundwork for the seamless integration of embodied skills and experiences into societal frameworks. This initiative not only aims for these advancements to become commonplace but also strives for their harmonious integration into the fabric of society. Looking forward, the “Moonshot” project vision extends to 2050, envisioning a society where individuals can seamlessly connect and collaboratively craft embodied experiences through CAs. Such a vision aligns with the UN SDGs 3, 10, and 16, underscoring the commitment to promoting health and well-being, reducing inequalities, and fostering peaceful and inclusive societies.

The Avatar Robot Cafe DAWN ver.β (https://dawn2021.orylab.com/en/) (Figures 6 and 7) serves as a highly successful showcase of the “Moonshot” project. This permanent experimental cafe offers a unique environment where individuals facing challenges in mobility can remotely operate avatar robots OriHime and OriHime-D from their homes or hospitals, thereby seamlessly providing services to patrons. These avatar robots serve as interactive digital extensions of human capabilities, made possible through sophisticated sensing and actuation technologies, as well as the fusion and fission of information. The design of DAWN aligns with Moore’s (2008) concept of “human enhancement” that entails efforts to surpass the current limitations of the human body through both natural and artificial means, either temporarily or permanently. DAWA exemplifies various forms of human augmentation as described by Raisamo et al. (2019): Augmented senses, Augmented action, and Augmented cognition.

Augmented senses: This enhancement extends the range of reality humans can perceive by integrating multisensory information.

Augmented action: This focuses on improving human physical abilities by sensing human actions and mapping them to actions in local, remote, or virtual contexts. Technological advancements such as artificial limbs, speech recognition, gaze controls, and teleoperation have enabled more precise functions, with robotics playing a significant role.

Augmented cognition: This refers to a form of human–system interaction that closely couples user and technology through physiological and neurophysiological sensing of the user's cognitive state. This paradigm aims to transform human–technology interaction by adapting user–system interactions in real-time based on cognitive state.

Figure 6.

Reception of cafe DAWN ver.β.

Figure 7.

Order delivery of avatar robot.

The remote operation of avatar robots at Cafe DAWN enhances user capabilities by enabling individuals with physical limitations to perform tasks and provide services in a physical space. The multisensory interfaces employed to control the robots augment sensory perception, offering users with limited mobility a more immersive and effective means of interacting with their environment. Additionally, the system's capacity to adapt to the cognitive state and operational needs of the users illustrates augmented cognition, optimizing the interaction between the user and the robotic system to ensure a seamless and efficient service experience.

A new paradigm

As augmented technology has rapidly evolved, the original framework has expanded to address the growing complexity and capabilities of human augmentation technologies. This evolution has led to the conceptualization of a new Human-Computer Integration paradigm, which not only includes augmented reality (AR) and augmented virtuality (AV) but also integrates essential components of wearable interactive technology with advancements in augmented senses, augmented action, and augmented cognition. To gain a comprehensive understanding of this emerging technology for effective research and design, Raisamo et al. (2019) extended Milgram and Kishino's reality-virtuality (RV) continuum (1994) to form the human augmentation technology continuum (Figure 8). Milgram and Kishino's RV continuum, introduced in 1994, has been instrumental in framing the development and understanding of virtual and AR technologies. This continuum spans from the completely real environment to the completely virtual environment, encompassing AR and AV in between. Figure 8 illustrates the extension of Milgram and Kishino's RV continuum by adding a y-axis representing the level of augmentation, which includes the number of employed sensors or tools and the integration level of augmented sense, augmented action, and augmented cognition.

Figure 8.

Human augmentation continuum (Raisamo et al., 2019).

The augmented human paradigm utilizes elements from AR, VR, ubiquitous computing, and other user interface paradigms, merging them in novel ways. This continuum reflects a broader spectrum of human enhancement possibilities, from sensory extensions and physical augmentations to cognitive enhancements. It enables a more comprehensive understanding of how technology can enhance human capabilities in diverse contexts.

Based on this human augmentation continuum, it is essential to identify which forms of human augmentation technologies are best suited to address specific user situations or problems. Additionally, understanding the social impacts and considerations associated with these technologies, both positive and negative, has become a critical area for further exploration. On the positive side, human augmentation technologies can significantly enhance quality of life by improving accessibility, increasing productivity, and enabling new forms of interaction and experience. For instance, AR can assist individuals with visual impairments by overlaying critical information in their FOV, while augmented action technologies can aid those with mobility issues in performing tasks that would otherwise be difficult or impossible. Augmented cognition can enhance learning and memory, providing significant benefits in educational and professional settings.

However, there are also potential negative social impacts to consider. Privacy concerns arise as augmented technologies often rely on extensive data collection and monitoring. There is also the risk of exacerbating social inequalities, as access to these advanced technologies may be limited to those who can afford them (Kapeller et al., 2021). Moreover, there may be psychological effects, such as dependency on technology or a diminished sense of personal achievement. Therefore, it is imperative to explore these issues comprehensively to ensure that the deployment of human augmentation technologies maximizes benefits while minimizing potential harms. This involves not only technological development but also ethical, social, and policy considerations to guide the responsible use of these innovations.

Opportunity and challenges

From an innovation adoption perspective, user and societal perceptions of new and emerging technologies are crucial for future development. From the user's view, Whitman (2018) conducted a survey to understand the characteristics of individuals who support or oppose human enhancement technologies. The study investigated five types of human enhancements: (1) Therapeutic use to restore ability, (2) Prevention with known risk or relevant family history, (3) Prevention without known risk or family history, (4) Enhancement beyond normal human ability, and (5) Enhancement significantly beyond normal.

The survey began with an assessment of technologies aimed at correcting health deficiencies and progressed to capture public opinion on using these technologies to enhance normal human abilities. Results showed that 96% of respondents supported vision enhancements as therapeutic interventions or preventive measures. However, support dropped significantly when these enhancements were elective or exceeded normal human capabilities. Additionally, over 95% of respondents agreed that technological aids, such as medications for cognitive decline due to aging or illness, are appropriate as therapeutic interventions. With the increasing prevalence of an aging society, finding new ways to address age-related disabilities is imperative. Human augmentation technologies have the potential to enable better and longer independent living, addressing the urgent need for innovative solutions to cope with and combat age-related disabilities.

A similar conclusion was reached by Villa et al. (2023), who conducted two scenario-based studies with 506 participants from four countries. Their research indicated that respondents were more positive about augmentations aimed at compensating for a missed skill. This study also suggested that different types of augmentations elicit varying levels of concern; cognitive augmentations were less well-received than motor augmentations, and sensory augmentations raised higher privacy concerns; Rousi and Renko (2020) examined cognitive enhancement from the body's perspective, categorizing integration levels as Endo (in-body), Exo (wearable/embodied), and External (environment).

In Particular, their study addressed users’ emotional attitudes toward human augmentation technologies, wearable devices, smart clothing, smart glasses, and cognitive enhancement games. They found that people are less willing to use brain or eye implants compared to smart glasses or smart textiles, suggesting that the level of integration significantly impacts augmentation acceptance. Collectively, these studies highlight the nuanced perceptions of human augmentation technologies and underline the importance of developing targeted and acceptable solutions.

From a societal perspective, it is widely acknowledged that public perception of a technology can significantly shape its development and adoption. Social acceptability is the most well-known and extensively studied factor influencing the adoption of emerging technologies (Lu et al., 2022; Muehlhaus et al., 2022). Although human augmentation can enhance the user's quality of life (Menuz et al., 2013), concerns about its broader societal impacts persist (Bavelier et al., 2019). Lack of social acceptability can affect the self-perception of augmented users and deter early adopters from embracing new technologies (Koelle et al., 2017). Factors influencing public attitudes towards human augmentation include social manipulation, accessibility, autonomy, and safety (Fitz et al., 2014; Harborth, 2022; Ho et al., 2022).

Concerns about social manipulation stem from the ability of augmented sensing technologies to generate fabricated information, potentially subjecting users to the influence of large corporations and compromising the credibility of information (Maras & Alexandrou, 2019).

Accessibility is another significant concern, as the high cost of these technologies may make them unattainable for socially vulnerable groups, such as the impaired or elderly, who stand to benefit the most from them (Obrenovic et al., 2007).

Autonomy can be compromised when technologies such as sensory sharing encroach on personal independence (Wolf et al., 2016). The ethical complexities are particularly pronounced with cognitive augmentations, as neurotechnological implants and chemical stimulants carry risks of adverse side effects and potential for misuse, including in malicious activities such as terrorism (Harborth, 2022; Ho et al., 2022; Tennison & Moreno, 2012).

Public concerns about safety are predominantly centered on the extensive reach of augmentation technologies, including big data, sensors, and surveillance systems. These technologies have the potential to exert unprecedented control over individuals, fueling fears of invasive monitoring and privacy erosion. The unauthorized access to large volumes of personal data could also result in identity theft and other forms of cybercrime (Davies et al., 2015; Khamis and Alt, 2021; Raisamo et al., 2019)

Nevertheless, human augmentation symbolizes a fresh technological paradigm brimming with vast potential and offering boundless opportunities for innovation. Yet, it also signifies a venture into uncharted territory, and overlooking potential adverse effects on individuals and society could precipitate undesirable outcomes in the future.

Conclusion

In 1964, McLuhan's theory of the “extension of man” posited that technology and media serve as extensions of human faculties, enhancing our ability to interact with and perceive the world. Each technological advancement extends a part of the human body or mind, thereby transforming societal structures and personal experiences. Interestingly, McLuhan's concept from decades ago resonates with the cutting-edge developments in human augmentation today. Artificial enhancements—such as prosthetics, wearable technology, and cognitive implants—embody McLuhan's idea by extending human capabilities beyond their natural limits. These modern augmentations not only supplement human abilities but also redefine the boundaries of what it means to be human.

Reflecting on the intricacies of haptic design and human augmentation, as elucidated by Dr. Minamizawa's keynote address, one is drawn into a realm where technology and human experience intertwine, promising transformative possibilities for our engagement with the world. The journey into haptic design reveals a rich tapestry of sensory exploration, where the nuances of touch are dissected and reconstructed to bridge physical and virtual realms. The concept of telexistence further expands these horizons, offering a glimpse into a future where physical presence transcends spatial limitations. Through robotic avatars and virtual interfaces, users navigate remote environments with a SoA and ownership, challenging traditional notions of presence and interaction.

Moreover, the convergence of haptic design with telexistence and human augmentation unveils a new frontier in sensory enhancement and bodily awareness. By integrating tactile feedback into prosthetics and wearable devices, Dr. Minamizawa and his peers propel us towards a deeper connection with our augmented selves, blurring the boundaries between biology and technology. This symbiotic relationship between humans and machines heralds not just enhanced capabilities but a reimagining of what it means to be embodied in an increasingly digital world.

Yet, amidst these advancements lies a call for deeper understanding and ethical consideration. As we delve into the realms of self-location, agency, and bodily ownership, we confront questions of identity and perception in virtual and remote contexts. The exploration of user interfaces and avatar morphology underscores the importance of design choices in shaping our experiences, urging us to tread carefully as we navigate this ever-expanding landscape. The journey towards human augmentation is not without its challenges. Perceptions of these technologies vary widely, from therapeutic interventions to enhancements significantly beyond normal human abilities. Understanding these nuanced perspectives is crucial for the responsible development and deployment of human augmentation technologies. Moreover, societal concerns about privacy, accessibility, autonomy, and safety underscore the need for ethical, social, and policy considerations to guide their implementation.

In navigating these opportunities and challenges, it becomes evident that telexistence and human augmentation represent not just technological advancements but also societal transformations. By embracing a human-integration approach and fostering inclusive dialogue, we can ensure that the future of human augmentation is one of empowerment, equity, and collective progress.

Footnotes

Declaration of conflicting interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Jiunde Lee

References

Andersen, R. A., Snyder, L. H., Li, C. S., & Stricanne, B. (1993). Coordinate transformations in the representation of spatial information. Current Opinion in Neurobiology, 3(2), 171–176. https://doi.org/10.1016/0959-4388(93)90206-E

Angelaki

D. E.

Cullen

K. E.

(2008). Vestibular system: The many facets of a multimodal sense. Annu Rev Neurosci 31, 125–150. https://doi.org/10.1146/annurev.neuro.31.060407.125555

Bavelier, D., Savulescu, J., Fried, L., Friedmann, T., Lathan, C., Schürle, S., & Beard, J. (2019). Rethinking human enhancement as collective welfarism. Nature, 3(3), 204–206. https://doi.org/10.1038/s41562-019-0545-2

Blanke

Metzinger

(2009). Full-body illusions and minimal phenomenal selfhood. Trends in Cognitive Sciences, 13(1), 7–13. https://doi.org/10.1016/j.tics.2008.10.003

Botvinick

Cohen

(1998). Rubber hands ‘feel’ touch that eyes see. Nature, 391(6669), 756. doi: 10.1038/35784

Braun, N., Debener, S., Spychala, N., Bongartz, E., Sörös, P., Müller, H. H. O., & Philipsen, A. (2018). The senses of agency and ownership: A review. Frontiers in Psychology, 16(9), 535. https://doi.org/10.3389/fpsyg.2018.00535

Cheng

Tian

J. F.

Kohno

., & Roesner, F. (2023) Exploring user reactions and mental models towards perceptual manipulation attacks in mixed reality. 32nd USENIX Security Symposium.

Cornelio, P., Haggard, P., Hornbaek, K., Georgiou, O., Bergström, J., Subramanian, S., & Obrist, M. (2022). The sense of agency in emerging technologies for human–computer integration: A review. Frontiers in Neuroscience, 16, 949138. https://doi.org/10.3389/fnins.2022.949138

Davies

Friday

Clinch

, etal (2015) Security and privacy implications of pervasive memory augmentation. IEEE Pervasive Computing 14(1): 44–53. https://doi.org/10.1109/MPRV.2015.13

10.

Dourish

. (2001) Where the Action Is: The Foundations of Embodied Interaction. MIT press. https://doi.org/10.7551/mitpress/7221.003.0006

11.

Ehrsson

H. H.

(2012). The Concept of Body Ownership And Its Relation To Multisensory Integration . MIT Press. https://doi.org/10.7551/mitpress/8466.003.0067

12.

Engelbart

. (1968). The Mother of All Demos. Last retrieved on 03/05/2024 from https://www.dougengelbart.org/mousesite/1968Demo.html

13.

Fitz, F. S., Nadler, R., Manogaran, P., Chong, E. W., & Reiner, P. B. (2014). Public attitudes toward cognitive enhancement. Neuroethics, 7(2), 173–188. https://doi.org/10.1007/s12152-013-9190-z

14.

Gibson, J. J. (1962). Observations on active touch. Psychological Review, 69(6), 477–491. https://doi.org/10.1037/h0046962

15.

Gorisse, G., Christmann, O., Houzangbe, S., & Richir, S. (2019). From robot to virtual doppelganger: Impact of visual fidelity of avatars controlled in third-person perspective on embodiment and behavior in immersive virtual environments. Frontiers in Robotics and AI, 6, 8. https://doi.org/10.3389/frobt.2019.00008

16.

Guterstam

Abdulkarim

Ehrsson

H. H.

(2015). Illusory ownership of an invisible body reduces autonomic and subjective social anxiety responses. Scientific Reports, 5(1), 9831. https://doi.org/10.1038/srep09831

17.

Hagura

Takei

Hirose

, etal (2007). Activity in the posterior parietal cortex mediates visual dominance over kinesthesia. Journal Neuroscience, 27(26): 7047–7053. https://doi.org/10.1523/JNEUROSCI.0970-07.2007

18.

Hanamitsu

Nakamura

., Nakatani

, & Minamizawa, K. (2015). Twech: a mobile platform to search and share visuo-tactile experiences. SIGGRAPH Asia 2015 Emerging Technologies, 1–2. https://doi.org/10.1145/2818466.2818495

19.

Harborth

(2022). Human autonomy in the era of augmented reality - A roadmap for future work. Information, 13(6), 289. https://doi.org/10.3390/info13060289

20.

Henriques

D. Y.

Cressman

E. K.

(2012). Visuomotor adaptation and proprioceptive recalibration. Journal of Motor Behavior, 44(6), 435–444. https://doi.org/10.1080/00222895.2012.659232

21.

Ho, B. Q., Otsuki, M., Kishita, Y., Kobayakawa, M., & Watanabe, K. (2022). Human augmentation technologies for employee well-being: A research and development agenda. International Journal Environment Resource Public Health, 19(3), 1195. https://doi.org/10.3390/ijerph19031195

22.

Huang, H. C., Lee, Y. T., Chen,W. Y., & Liang, C. (2017). The sense of 1PP-location contributes to shaping the perceived self-location together with the sense of body-location. Frontiers in Psychology, 8, 236543. https://doi.org/10.3389/fpsyg.2017.00370

23.

Kamijo

Minamizawa

Koyanagi

. (2016). Development of Unmanned Multi Track Robot Controlled by Remote Operation System. Last retrieved on May 20, 2024 from https://www.jstage.jst.go.jp/article/geosocabst/2016/0/2016_002/_pdf/-char/ja

24.

Kapeller

Felzmann

Fosch-Villaronga

, etal (2021) Implementing ethical, legal, and societal considerations in wearable robot design. Applied Sciences, 11(15), 6705. https://doi.org/10.3390/app11156705

25.

Kappenman

E, S.

Farrens

J, L.

Zhang

., Stewart A, X., & Luck, S, J. (2021). ERP CORE: An open resource for human event-related potential research. NeuroImage 225. 117465. https://doi.org/10.10.1016/j.neuroimage.2020.117465

26.

Kheddar

(2001). Teleoperation based on the hidden robot concept. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, 31(1): 1–13. https://doi.org/10.1109/3468.903862

27.

Kheddar

Chellali

Coiffet

. (2014). Virtual environment-assisted teleoperation. In S. H. Kelly & M. S. Kay (Eds.), Handbook of vrtual environments: Design, implementation, and application. CRC Press, 1109–1144. https://doi.org/10.1201/b17360-55

28.

Kilteni

Groten

Slater

(2012). The sense of embodiment in virtual reality. Presence: Teleoperators and Virtual Environments, 21, 373–387. https://doi.org/10.1162/PRES_a_00124

29.

Kilteni, K., Maselli, A., Kording, K. P., & Slater, M. (2015). Over my fake body: Body ownership illusions for studying the multisensory basis of own-body perception. Frontiers in Human Neuroscience, 9, 119452. https://doi.org/10.3389/fnhum.2015.00141

30.

Kocaballi

A. B.

Laranjo

Coiera

(2019). Understanding and measuring user experience in conversational interfaces. Interacting with Computers, 31(2), 192–207. https://doi.org/10.1093/iwc/iwz015

31.

Koelle, M., Ali, A. E., Cobus, V., Heuten,W., & Boll, S. C. (2017). All about acceptability? Identifying factors for the adoption of data glasses. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, (CHI ‘17), Denver, Colorado, USA (pp. 295–300). Association for Computing Machinery. https://doi.org/10.1145/3025453.3025749

32.

Kokkinara

Slater

(2014). Measuring the effects through time of the influence of visuomotor and visuotactile synchronous stimulation on a virtual body ownership illusion. Perception 43(1): 43–58. https://doi.org/10.1068/p7545ap

33.

Lenggenhager

Mouthon

Blanke

(2009). Spatial aspects of bodily self-consciousness. Consciousness and Cognition, 18(1), 110–117. https://doi.org/10.1016/j.concog.2008.11.003

34.

Lewis

(1998). Designing for human-agent interaction. AI Magazine, 19(2), 67–67.

35.

Lipkie

E, et al. (1982). Star graphics. An object-oriented implementation. Computer Graphics 16(3): 115–124. https://doi.org/10.1145/965145.801270

36.

Longo

Matthew R

Schüür

Friederike

Kammers

Marjolein P.M

, etal (2008) What is embodiment? A psychometric approach. Cognition 107(3): 978–998. https://doi.org/10.1016/j.cognition.2007.12.004

37.

Lopez-Sola

Moreno-Bote

Arsiwalla

X. D.

(2021). Sense of agency for mental actions: Insights from a belief-based action-effect paradigm. Consciousness and Cognition, 96, 103225. https://doi.org/10.1016/j.concog.2021.103225

38.

Lopez

Halje

Blanke

(2008). Body ownership and embodiment: Vestibular and multisensory mechanisms. Neurophysiologie Clinique/Clinical Neurophysiology, 38(3), 149–161. https://doi.org/10.1016/j.neucli.2007.12.006

39.

Thomaz

Epstein

D. A.

(2022). Understanding people’s perceptions of approaches to semi-automated dietary monitoring. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 6(3), 1–27. https://doi.org/10.1145/3550288

40.

Lugrin

J. L.

Latt

Latoschik

M. E.

(2015). Avatar anthropomorphism and illusion of body ownership in VR. In 2015 IEEE virtual reality (VR) (pp. 229–230). https://doi.org/10.1109/VR.2015.7223379

41.

Macy

M, W.

Willer

. (2002). From factors to actors: Computational sociology and agent-based modeling. Annual Review of Sociology 28(1): 143–166. https://doi.org/10.1146/annurev.soc.28.110601.141117

42.

Maras

M, H.

Alexandrou, A . (2019). Determining authenticity of video evidence in the age of artificial intelligence and in the wake of Deepfake videos. The International Journal of Evidence and Proof 23(3): 255–262. https://doi.org/10.1177/1365712718807226

43.

McCloud

(1993). Understanding Comics: The Invisible Art. Kitchen Sink Press. Chicago / Turabian.

44.

McLuhan

(1964). Understanding Media - The Extensions of Man. McGraw-Hill.

45.

Menuz

Hurlimann

Godard

(2013). Is human enhancement also a personal matter? Science and Engineering Ethics, 19(1), 161–177. https://doi.org/10.1007/s11948-011-9294-y

46.

Milgram

Kishino

(1994). A taxonomy of mixed reality visual displays. IEICE Trans. Information System, 77, 1321–1329. https://doi.org/10.1016/0141-9382(84)90044-1

47.

Minamizawa

Kakehi

Nakatani

., Mihara S., & Tachi, S. (2012) TECHTILE toolkit: A prototyping tool for design and education of haptic media. Proceedings of the 2012 Virtual Reality International Conference, 26: 1–2. https://doi.org/10.1145/2331714.2331745

48.

Moore

(2008). Enhancing Me: The Hope And The Hype Of Human Enhancement. John Wiley and Sons. https://doi.org/10.1038/npre.2008.2139.1

49.

Muehlhaus

Steimle

Koelle

(2022). Feather hair: Interacting with sensorized hair in public settings. In Designing Interactive Systems Conference (DIS ‘22), Virtual Event, Australia (pp. 1228–1242). Association for Computing Machinery. https://doi.org/10.1145/3532106.3533527

50.

Nakano, K., Isoyama, N., Monteiro, D., Sakata, N., Kiyokawa, K., & Narumi, T. (2021). Head-mounted display with increased downward field of view improves presence and sense of self-location. IEEE Transactions on Visualization and Computer Graphics, 27(11), 4204–4214. https://doi.org/10.1109/TVCG.2021.3106513

51.

Nishio

Watanabe

Ogawa

., & Ishiguro H.

(2012). Body Ownership Transfer to Teleoperated Android Robot. In S.S. Ge, O. Khatib, J.J. Cabibihan, R. Simmons, M.A. Williams (Eds.), Social Robotics. ICSR 2012, Lecture Notes in Computer Science, 7621. Heidelberg: Springer. https://doi.org/10.1007/978-3-642-34103-8_40

52.

NTT Docomo . (2023-05-21). Feeltech™. Last retrieved on May 21, 2024 from https://www.docomo.ne.jp/special_contents/brand/society/07/

53.

Obrenovic

Abascal

Starcevic

(2007). Universal accessibility as a multimodal design issue. Communication ACM, 50(5), 83–88. doi: 10.1145/1230819.1241668

54.

Peirce

C. S.

(1894). What Is a Sign?. Published in part in CP 2.281, 285, and 297–302, and in full in EP 2:4–10. Peirce Edition Project.

55.

Petkova

V. I.

Khoshnevis

Ehrsson

H. H.

(2011). The perspective matters! multisensory integration in egocentric reference frames determines full-body ownership. Frontiers in Psychology, 2, 35. doi: 10.3389/fpsyg.2011.00035

56.

Pfeiffer

, et al. (2013). Multisensory origin of the subjective first-person perspective: Visual, tactile, and vestibular mechanisms. PLOS ONE, 8, e61751. doi: 10.1371/journal.pone.0061751

57.

Piryankova

I. V.

Wong

H. Y.

Linkenauger

S. A.

, etal (2014). Owning an overweight or underweight body: Distinguishing the physical, experienced and virtual body. PLoS ONE, 9(8), e103428. doi: 10.1371/journal.pone.0103428

58.

Raisamo

Rakkolainen

Majaranta

, etal (2019). Human augmentation: Past, present and future. International Journal of Human-Computer Studies, 131, 131–143. doi: 10.1016/j.ijhcs.2019.05.008

59.

Rogers

. (1989) Icons at interface: Their usefulness. Interacting with Computers 1(1): 105 –117. doi: 10.1016/0953-5438(89)90010-6

60.

Rousi

Renko

(2020). Emotions toward cognitive enhancement technologies and the body–attitudes and willingness to use. International Journal of Human-Computer Studies, 143(1), 102472. doi: 10.1016/j.ijhcs.2020.102472

61.

Salomon

Lim

Pfeiffer

, etal (2013). Full body illusion is associated with widespread skin temperature reduction. Frontiers in Behavioral Neuroscience, 7. doi: 10.3389/fnbeh.2013.00065

62.

Saraiji

M Y

Fernando

Minamizawa

(2015) HUG Project. Last retrieved on May 15, 2024 from https://www.embodiedmedia.org/projects/hug-project

63.

Serino

Alsmith

Costantini

, etal (2013). Bodily ownership and self-location: Components of bodily self-consciousness. Consciousness and Cognition, 22(4), 1239–1252. doi: 10.1016/j.concog.2013.08.013

64.

Shibasaki

Kamiyama

Czech

, Obata K., et al. (2020). Interest arousal by haptic feedback during a storytelling for kindergarten children. In I. Nisky, J. Hartcher-O’Brien, M. Wiertlewski, & J. Smeets (Eds.), Haptics: Science, Technology, Applications. EuroHaptics 2020, 12272, 518-526. Cham., Springer. doi: 10.1007/978-3-030-58147-3_57

65.

Shibasaki

Kamiyama

Minamizawa

(2016-10) Designing a haptic feedback system for hearing-impaired to experience tap dance. Adjunct Proceedings of the 29th Annual ACM Symposium on User Interface Software and Technology: 97–99. doi: 10.1145/2984751.2985716

66.

Shneiderman

(1982) The future of interactive systems and the emergence of direct manipulation. Behaviour and Information Technology 1(3): 237–256. doi: 10.1080/01449298208914450

67.

Slater

Perez-Marcos

Ehrsson

H H

, etal (2008). Towards a digital body: the virtual arm illusion. Front Hum Neuroscience, 2, 6. doi: 10.3389/neuro.09.006.2008

68.

Tennison

M N

Moreno

J D

(2012) Neuroscience, Ethics, and National Security: The State of the Art. PLoS Biol, 10(3), e1001289. doi: 10.1371/journal.pbio.1001289

69.

Tsakiris

Prabhu

Haggard

(2006). Having a body versus moving your body : How agency structures body-ownership. Conscious Cognition 15(2): 423–432. doi: 10.1016/j.concog.2005.09.004.

70.

Tsakiris

Manos

Prabhu

Gita

Haggard

Patrick

(2006) Having a body versus moving your body: How agency structures body-ownership. Consciousness and Cognition, 15(2): 423–432. doi: 10.1016/j.concog.2005.09.004

71.

Villa

Niess

Nakao

Lazar

Schmidt

Machulla

T. K.

(2023). Understanding perception of human augmentation: A mixed-method study. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ‘23), Article 65 (pp. 1–16). Association for Computing Machinery.

72.

Waltemate

Gall

Roth

, etal (2018). The impact of avatar personalization and immersion on virtual body ownership, presence, and emotional response. IEEE Transactions on Visualization and Computer Graphics, 24(4), 1643–1652. doi: 10.1109/TVCG.2018.2794629

73.

Whitman

Love

Rainville G

etal (2018). U.S. Public Opinion And Interest On Human Enhancements Technology. AARP

74.

Wolf

Grodzinsky

Miller

(2016). There's something in your eye: Ethical implications of augmented visual field devices. Journal of Information Communication Ethics Society, 14(3), 214–230. doi: 10.1108/JICES-10-2015-0035

75.

Yuan

Steed

(2010) Is the rubber hand illusion induced by immersive virtual reality?. IEEE Virtual Reality Conference (VR): 95–102. doi: 10.1109/VR.2010.5444807

Bridging Divide with Innovative Media: Telexistence and Human Augmentation

Abstract

Keywords

Introduction

Haptic design

Telexistence

Sense of self-location

Sense of agency

Bodily ownership

Human augmentation

Not a “Tool” but an “Integration”

A new paradigm

Opportunity and challenges

Conclusion

Footnotes

Declaration of conflicting interests

Funding

ORCID iD

References