Haptic Technology for Hearing Loss: A Systematic Review of Technical Feasibility,Usability,and User Experience

Abstract

Nearly 27 million Americans aged 50 and up experience hearing loss, with varying degrees and types of impairment. Haptic technology offers promising support for these individuals, either as standalone solutions, complements to existing devices, or alternatives for those not benefitting from traditional devices. This systematic review investigates the technical feasibility, usability, and user experience of haptic technology for hearing assistance. Using the PRISMA method, eight relevant articles were identified. Results indicated a focus on technical feasibility, with positive outcomes for direction of arrival, sound localization, and sensory substitution. Some studies also investigated usability and user experience, exploring user preferences, social acceptance, and integration with existing devices. The limited studies constrain the ability to draw substantial conclusions. Nevertheless, this review offers insights to guide future device development, highlighting the need for more research in usability and user experience alongside technical advancements.

Among the human senses, hearing plays an essential role in human cognitive and perceptual functions (Lotto & Holt, 2011). About 20% of the global population lives with hearing loss. However, the adoption of hearing devices remains low, with only one in seven individuals opting for hearing aids (World Health Organization, 2024). The low adoption rates can be attributed to various factors, including cost, social stigma, technological confusion, and the inability to address all facets of hearing loss (Gallagher & Woodside, 2018).

Researchers have explored leveraging haptic technology as an additional or alternative aid for those with hearing loss (Flores Ramones & Del-Rio-Guerra, 2023). Haptic devices use touch sensations to convey information, often through vibrations or other tactile feedback (Culbertson et al., 2018). These devices can function independently or be integrated with existing solutions, such as hearing aids and cochlear implants (Fletcher, 2020). Haptic devices can provide hearing support that traditional hearing aids cannot offer: (1) They relay crucial information (e.g., fire alarms), which may be missed when traditional hearing aids are removed (e.g., during sleep); (2) They enhance sound awareness, especially from behind, where traditional hearing aids often fall short (Flores Ramones & Del-Rio-Guerra, 2023); (3) Their hardware can be integrated into everyday accessories (e.g., watches), making them more acceptable (Sakuma et al., 2019); and (4) They are customizable, allowing users to tailor feedback patterns to their needs.

Previous reviews have summarized technical developments in haptic devices intended for those with hearing loss (Flores Ramones & Del-Rio-Guerra, 2023). However, there is little analysis and discussion of the usability and perceptual aspects of their implementation. This systematic review aims to address this gap by examining (1) Technical feasibility: identifying technical features and limitations of haptics to improve auditory perception, (2) Usability: investigating the cognitive demands, integration, and overall usability of haptic devices in a user’s daily life, and (3) User experience: highlighting user preferences and feedback regarding haptic devices.

Utilizing the PRISMA method, a search was conducted across Engineering Village, PubMed, Scopus, and Web of Science in October 2023. This search was conducted using a set of keywords related to the topic of haptic technology and hearing loss. The search string included these parts: haptic-related—“haptic or tactile or sensory feedback,” hearing-impairment-related—“hearing impaired or hearing loss,” and user-related—“user experience or usability or evaluation or assessment.” The search was limited to English-language peer-reviewed publications, including journal articles and conference papers published between 2018 and 2023. Inclusion criteria were: (1) Studies evaluating haptic devices for hearing impairment, (2) Assessment of technical feasibility, and (3) Inclusion of user feedback or usability evaluations. Eight relevant articles were identified for analysis.

The analysis of the eight studies revealed findings that aligned with the three initial focus areas: Technical Feasibility, Usability, and User Experience.

Technical Feasibility

Sound Localization and Spatial Awareness

Five studies demonstrated that haptic devices could improve sound localization and spatial awareness by providing feedback to the user about the direction of a sound stimulus through vibrations. In Michaud et al. (2022), the researchers evaluated the performance of a haptic belt by playing test sounds at different speaker locations and then comparing the predicted and reference Direction of Arrival (DoA) angles. The results confirmed accurate DoA estimation. However, the accuracy of the directional information was reduced in noisy environments or with low-frequency sounds.

Speech and Sound Intelligibility

Four studies focused on tasks that tested participants’ ability to identify and understand vibrational patterns translated from sounds (e.g., speech, dog bark, smoke alarm, door knock, phone ringing). Three of these studies included a training session for participants, which was crucial for their success (Fletcher et al., 2019; Goodman et al., 2020; Otoom et al., 2019). However, two of the studies did not directly measure the before-and-after effects of training (Goodman et al., 2020; Otoom et al., 2019). In contrast, the one study that measured training effects (Fletcher et al., 2019) tested speech intelligibility during a speech-in-noise task with and without haptic stimulation of the wrist, both before and after a 20-minute training session. Initially, haptic stimulation showed no significant impact, but after training, speech intelligibility improved. In the fourth study, Perrotta et al. (2021) conducted a longitudinal study over 1 month, including various speech and sound intelligibility tasks. The accuracy of performance improved as time progressed. Although the other studies (Goodman et al., 2020; Otoom et al., 2019) did not measure pre- and post-training effects, their findings suggest that training is essential for effectively using haptic feedback in sound identification tasks.

Usability

Cognitive Workload and Attention

During four experiments, participants commented on their perceived mental effort while relying on haptic feedback to alert them of sounds in their environment (e.g., vibrational navigation cues while driving, haptic captioning to watch a video, sound feedback in public settings). For example, Wang et al. (2023) had participants wear a haptic vibrator on their wrist (haptic captioning), watch a short video, and mark when they identified a speaker voice transition. While there was high accuracy of 93.75%, there were concerns about mental effort. One participant stated, “If I have to wear it for a long time, I will likely have sensory overload.” There were also concerns about it being distracting and adding more fatigue instead of functioning as an intended aid (Goodman et al., 2020).

Integration with Existing Assistive Technologies

The topic of integrating haptic devices with existing assistive devices (e.g., hearing aids and cochlear implants) was brought up in three studies. One study, Fletcher et al. (2019), measured the speech-in-noise performance of a wrist-worn haptic device with a cochlear implant. Participants reported that the additional haptic input provided valuable cues that enhanced their overall understanding of speech. While the other two studies did not directly measure usability, the topic of integration was discussed with participants. In Goodman et al. (2020), participants mentioned limitations of their existing hearing aids and cochlear implants, such as difficulty discriminating between sounds, inadequate background filtering, and poor speech comprehension, suggesting a need for a complementary device to help with these limitations. Based on the positive results of the neckband device in sound localization accuracy, researchers in Sakuma et al. (2019) noted that a haptic device could supplement auditory cues for people with hearing aids.

User Experience

User Comfort and Acceptance

Researchers explored different form factors and design elements, with three studies adding an exit interview to better understand user preferences and experience, and social acceptability. In Goodman et al.’s (2020) smartwatch study, 15 out of 16 participants reported feeling comfortable using the device in public settings. They commented about not caring what others thought and were excited to show it off. There was a trend in form factor design, with five out of the eight studies experimenting with a wrist-worn haptic device. Otoom et al. (2019) explored participants’ preferences for a device during a driving scenario (e.g., gloves and pillow); the preference resulted in the bracelet design. Participants desired an experience where they could customize the haptic feedback to fit their needs while also helping users distinguish the difference between patterns (reminders, media content, and alarms; Goodman et al., 2020; Wang et al., 2023).

Without directly being asked or tested on this scenario, a common use case emerged, with all participants in Goodman et al.’s (2020) study mentioning potential benefits for using the device for safety in the home (alarms, baby crying, and door being broken down). This further highlights the need for customization and a focus on including the user in the design.

Future research may focus on the following areas to understand further and optimize the use of haptic devices for people with hearing loss: (1) Longitudinal studies: Investigate the long-term integration of haptic devices into the daily lives of users to assess sustained usability and effectiveness; (2) Cognitive load measurement: Quantify the cognitive load associated with haptic feedback to determine how distracting these devices truly are in various scenarios; (3) Comparative studies: Conduct comparative studies of haptic devices versus traditional hearing aids in real-world scenarios to identify their unique use cases and strengths; and (4) Potential for mild hearing loss: Explore the potential of haptic devices for individuals with mild hearing loss who may not yet require traditional hearing aids, assessing their effectiveness and user acceptance.