Abstract
This study investigated whether adding auditory angular and curved sounds to tactile angle and curve shapes – one unspecified sound to one unspecified shape – positively influences the accuracy and exploration time in recognising tactile angles and curves when experienced and inexperienced in using haptic touch. A within-participant experiment was conducted, with two groups of participants: experienced and inexperienced in using haptic touch, and with two conditions: congruous (e.g., angle shape and angular sound) and incongruous (e.g., angle shape and curved sound) tactile and auditory shape information. Adding congruous auditory angular and curved sounds to tactile angle and curve shapes positively influences the accuracy in recognising tactile angles and curves both when experienced and inexperienced in using haptic touch, and the exploration time on correct recognitions when experienced. People integrate tactile and auditory (angle; curve) shape information and this improves their proficiency in recognising tactile angles and curves.
Keywords
Introduction
Haptic touch – the combination of touch and movement (Katz, 1989; Millar, 1997, 2008) – mixes up ≈70% of tactile two-dimensional (2D) angles and curves (Graven, 2016), which makes it difficult for people to recognise even the most common shapes. People may not recognise an illustration of the Pythagorean triangle, for example, because through haptic touch, they perceive the right angle as a curve. This angle-curve mix-up has serious consequences for people who rely on haptic touch, for example, people with total blindness: lack of awareness of important, sometimes life-saving information, such as ‘flammable’ and ‘poisonous’ on public signs. Miscommunication with sighted friends and family, for example, when exploring a picture or studying the details of a fire evacuation route. Shortcomings in school subjects, such as mathematics and science, where the teaching relies on diagrams, figures, graphs, and illustrations.
Previous research on recognition proficiency (i.e., accuracy, exploration time, or both) of haptic touch has taken two main directions. One strand has focused on what shape geometry is the least difficult to recognise by haptic touch alone (e.g., Crewdson & Zangwill, 1940; Davidson, 1972; Gordon & Morison, 1982; Graven et al., 2020; Hunter, 1954; Rubin, 1936; Voisin et al., 2002; Wijntjes & Kappers, 2007). For example, Wijntjes and Kappers (2007) have found that acute angles are easier to perceive as angles than obtuse angles, which are sometimes perceived as curves. The other main direction has focused on whether adding shape information to one or more additional senses influences the recognition proficiency of haptic touch (e.g., Graven, 2004, 2005; Graven & Desebrock, 2018; Lobb, 1965, 1970; Millar, 1971). Such research has mainly added visual shape information to the tactile shapes and has found that this improves the recognition proficiency of haptic touch (e.g., Graven, 2004, 2005; Lobb, 1965, 1970; Millar, 1971).
One previous research study (i.e., Graven & Desebrock, 2018) has investigated the proficiency of haptic touch in recognising tactile shapes when adding auditory shape information instead of visual shape information (and a few research studies have explored people’s ability to associate tactile and auditory shape information, match it, or both; for example, Fryer et al., 2014; Graven & Desebrock, 2018; Kassuba et al., 2013; Kim & Zatorre, 2010, 2011). Graven and Desebrock (2018) added auditory (non-word) angular and curved sounds, based on the spoken words ‘bouba’ (curved sound) and ‘kiki’ (angular sound), to tactile bouba (curve) and kiki (angle) shapes, and found that recognition accuracy improves but only when people are inexperienced, rather than experienced, in using haptic touch. In the task instructions, however, Graven and Desebrock (2018) specified what shapes were to be recognised before presenting them, that is, by asking their participants to, ‘Please, name the tactile shape and the audio either “bouba” or “kiki”: one name only’ (p. 205). Moreover, both the tactile shapes and the auditory sounds in Graven and Desebrock (2018) included several repetitions of the angles and curves: six and three, respectively, giving the participants plenty of opportunities to recognise them correctly. Consequently, the improvement in recognition accuracy could have resulted from the use of categorical (bouba; kiki) information in the task instructions, a word-shape or word-sound-shape (instead of sound-shape) association by the participants, together with the repetition of angles and curves both in the tactile shapes and the auditory sounds or by this alone, rather than facilitative effects of auditory angular and curved sounds (see Fryer et al., 2014; Graven and Desebrock, 2018; Heller et al., 1996; Heller & Gentaz, 2014; Köhler, 1929; Pathak & Pring, 1989; Ramachandran & Hubbard, 2001).
This study, therefore, investigated whether adding auditory angular and curved sounds to tactile angle and curve shapes – one unspecified sound to one unspecified shape – positively influences the accuracy and exploration time in recognising tactile angles and curves when experienced and inexperienced in using haptic touch.
Method
Design
This study was designed as a within-participant experiment, with two groups of participants: experienced and inexperienced in using haptic touch, and with two conditions: congruous (e.g., angle shape and angular sound) and incongruous (e.g., angle shape and curved sound) tactile and auditory shape information. The Medical Sciences Inter-Divisional Research Ethics Committee, University of Oxford, provided approval for this study (Ref. No. MS-IDREC-C1-2015-200).
Participants
Twenty-five participants took part in this study and were compensated for their time. Thirteen participants were experienced in using haptic touch (7 females, mean age = 47.31 years): 11 were congenitally blind, and two were blinded less than four months after birth (see Sathian, 2000). Six had total blindness, three had light perception (perceiving a light source), and four had light projection (perceiving where a light source is situated: see World Health Organization [WHO], 2021). All were braille readers and all but one reported being experienced in using haptic touch to explore tactile pictures, figures, illustrations, and so on (see Table 1). Twelve participants were inexperienced in using haptic touch: they had normal/corrected-to-normal vision and were blindfolded for this study (see Sathian, 2000). They did not read braille and reported having very little or no experience in using haptic touch to explore tactile pictures, figures, illustrations, and so on (see Table 1). All 25 participants had normal/corrected-to-normal hearing. None of the participants possessed a cognitive delay or impairment, nor any physical disabilities and their education ranged from comprehensive school level to a doctoral degree level. All were naïve to the bouba/kiki-effect.
Self-reported experience in exploring tactile pictorial information.
Experience in exploring tactile pictures, diagrams, figures, graphs, illustrations, and symbols (without vision) produced on swell paper or plastic embossing film, or as a thermoform collage.
Test material
Tactile test material
Four angles and four curves, based on the Moon characters (in Graven, 2016; see Moon Literacy, 2021), were printed on swell paper (1.5 pt. line; height ≈ 0.5 mm) and glued on foam board (50 mm × 50 mm; 5-mm thick), making eight individual picture cards. The picture cards were presented on a (230 mm × 150 mm) rubber mat. (See Figures 1 and 2.)
Tactile test material.
Tactile shapes.
Auditory test material
Four angular and four curved sounds were created, using a sine-wave generator and pitch bend, making eight individual shape-melodies; that is, one shape-melody for each tactile (angle; curve) shape. All shape-melodies were based on the bouba and kiki (non-word) sounds created by Graven and Desebrock (2018; see also Graven & Desebrock, 2019). The curved shape-melodies were based on bouba, and the angular shape-melodies on kiki. (See Table 2.) The shape-melodies were presented via an MP3 player, with one built-in loudspeaker.
Auditory sounds.
Procedure
There were 16 trials: one tactile shape and one auditory sound per trial. The order of congruous and incongruous tactile and auditory (angle; curve) shape information was randomly assigned to the 16 trials (i.e., to eight trials each), as was the order of angles and curves.
The experiment took place in a quiet room with a neutral colour interior. Distinct light sources, such as lamps, were removed. All participants were tested individually. The participants who were blindfolded put on their blindfold before the set-up of the test material and kept their blindfold on throughout the entire experiment. (See Graven & Desebrock, 2019.))
Both the rubber mat and the MP3-player were placed directly in front of the participant, with the rubber mat closest to the participant. The rubber mat prevented the presented tactile picture card from moving around on the table. The MP3-player was placed on the other side of the rubber mat (i.e., between the rubber mat and the experimenter). The loudspeaker was positioned closest to the participant. All the participants were invited to explore the rubber mat (e.g., for size and texture) before the test material itself was introduced. (See Graven & Desebrock, 2019.)
Next, the experimenter explained that one tactile picture card would be placed on the rubber mat and that some auditory information would be played from the MP3-player, as soon as the participant started exploring the tactile information on the presented picture card. The experimenter also explained that the participant could explore the tactile information for as long as they needed to and that the auditory sound would appear one time only per tactile picture card.
Finally, the experimenter asked the participant to make a fist with both hands – to prevent them from feeling around – and, guided by the experimenter if needed, to place their fists on top of the presented tactile picture card, with the experimenter’s left hand being placed on top of the participants hands. (See Graven & Desebrock, 2019.) The experimenter then asked the participant to explore the tactile information on the presented picture card and to listen to the auditory information from the MP3-player to recognise the shape: to start when the experimenter removed their hand and to stop by saying ‘yes’, upon which the tactile picture card would be removed. Next, the experimenter removed their left hand from the participant’s fists and, as soon as the participant touched the tactile (angle; curve) shape on the presented picture card, the experimenter played the auditory (angular; curved) sound from the MP3-player. The experimenter did not respond to the participant’s answer.
Task
Please, tell me what shape the figure is (on the presented tactile picture card).
Scoring
The accuracy, or the number of correct recognitions of the tactile (angle; curve) shape.
The exploration time, or the number of seconds from when the experimenter’s left hand was removed from the participant’s fists to when the participant said ‘yes’.
Analysis
Accuracy
First, the mean recognition accuracy in the incongruous condition was calculated (for the experienced and inexperienced groups separately) to establish a point of comparison for the recognition accuracy when congruous auditory angular and curved sounds were added to the tactile angle and curve shapes.
Next, two chi-square goodness-of-fit tests (experienced; inexperienced) were conducted to test whether adding congruous auditory angular and curved sounds to tactile angle and curve shapes positively influenced the accuracy in recognising tactile angles and curves when (test 1) experienced, and (test 2) inexperienced in using haptic touch. The participants were categorised as ‘correct’ when their recognition accuracy was higher in the congruous condition than their group’s (experienced; inexperienced) mean recognition accuracy in the incongruous condition. If the added auditory angular and curved sounds positively influenced the accuracy of haptic touch in recognising tactile angles and curves, then the number of correct participants (experienced; inexperienced) should be significantly above the chance level (of 50%) when the tactile and the auditory (angle; curve) shape information was congruous.
Exploration time
The mean exploration time on correct recognitions in the incongruous condition was calculated (for the experienced and inexperienced groups separately). Again, this was intended to establish a point of comparison for the congruous condition, that is, for whether the exploration time on correct recognitions was positively influenced when congruous auditory angular and curved sounds were added to the tactile angle and curve shapes.
Finally, two chi-square goodness-of-fit tests (experienced and inexperienced) were conducted to test whether adding congruous auditory angular and curved sounds to tactile angle and curve shapes positively influenced the exploration time on correctly recognised tactile angles and curves when (test 1) experienced, and (test 2) inexperienced in using haptic touch. The participants were categorised as ‘fast’ when their mean exploration time on correct recognitions was lower in the congruous condition than their group’s (experienced; inexperienced) mean exploration time on correct recognitions in the incongruous condition. If the added auditory angular and curved sounds positively influenced the exploration time of haptic touch on correct recognitions, then the number of fast participants (experienced; inexperienced) should be significantly above the chance level (of 50%) when the tactile and the auditory (angle; curve) shape information was congruous.
Results
Accuracy
The participants who were experienced in using haptic touch recognised on average 3.54 (SD = 0.90) of the eight tactile (angle; curve) shapes in the incongruous condition, and the participants who were inexperienced recognised 3.25 (SD = 1.29). (See Figure 3). Hence, in the congruous condition, the participants who were experienced in using haptic touch were counted as ‘correct’ when correctly recognising ⩾4 of the eight tactile angles and curves, and those who were inexperienced when they correctly recognised ⩾3.
Correct recognition per participant and trial.
In the congruous condition, 12 of the 13 participants who were experienced in using haptic touch were counted as ‘correct’, and 10 of the 12 who were inexperienced in using it (see Figure 3) – the number of correct participants was significantly above the chance level (of 50%) in both groups: χ2(1. N = 13) = 9.31, p = .00 and χ2(1. N = 12) = 5.33, p = .02, respectively. Adding congruous auditory angular and curved sounds to tactile angle and curve shapes positively influenced the accuracy in recognising tactile angles and curves, both when experienced and inexperienced in using haptic touch.
Exploration time
On average, the participants who were experienced in using haptic touch explored each one of their correctly recognised tactile (angle; curve) shapes in the incongruous condition for 9.75 s (SD = 7.20), and the participants who were inexperienced for 13.12 s (SD = 5.52). (See Table 3.) So, in the congruous condition, the participants who were experienced in using haptic touch were counted as ‘fast’ when their mean exploration time on correctly recognised tactile angles and curves was ⩽9.8 s, and those who were inexperienced when their mean exploration time was ⩽13.1 s.
Mean exploration time per participant on correct recognitions.
Missing value on Trial 1. Imputed value = F’s own mean exploration time on correct recognitions in the congruous condition.
In the congruous condition, 10 of the 13 participants who were experienced in using haptic touch were counted as ‘fast’, and six of the 12 who were inexperienced in using haptic touch (see Table 3) – the number of fast participants who were experienced significantly above the chance level (of 50%): χ2(1. N = 13) = 3.77, p = .05, and the number of fast participants who were inexperienced at the chance level: χ2(1. N = 12) = 0.00, p = 1.00. Adding congruous auditory angular and curved sounds to tactile angle and curve shapes positively influenced the exploration time on correctly recognised tactile angles and curves when experienced in using haptic touch, but not when inexperienced.
Discussion
This study found that adding congruous auditory angular and curved sounds to tactile angle and curve shapes – one unspecified sound to one unspecified shape – positively influenced the accuracy in recognising tactile angles and curves both when experienced and inexperienced in using haptic touch, and the exploration time on correct recognitions when experienced. People integrate tactile and auditory (angle; curve) shape information and this improves their proficiency in recognising tactile angles and curves.
These results somewhat contradict Hötting and Röder’s (2009) suggestion that ‘the likelihood for multisensory integration might be lower in the blind than in the sighted because of their enhanced perceptual skills within the tactile and auditory modality’ (p. 169). According to Hötting and Röder’s (2009) suggestion, the participants who were inexperienced in using haptic touch in this study should integrate the congruous tactile and auditory (angle; curve) shape information, and they did, whereas those who were experienced in using haptic touch should not, and yet they did in this study.
However, Hötting and Röder (2009) based their suggestion on a previous research study (i.e., Hötting & Röder, 2004). In this research study, Hötting and Röder (2004) investigated people’s proficiency in judging the number of perceived tactile stimuli, that is, using metallic pins presented on their finger-tip together with or without task-irrelevant auditory sounds (i.e., sinusoidal tones), and not on people’s proficiency in recognising tactile (angle; curve) shapes when using haptic touch and when auditory (angular; curved) sounds were added to the tactile (angle; curve) shapes. Indeed, the task set in Hötting and Röder’s (2004) study and the study undertaken herein was different, namely, counting the number of metallic pins and recognising tactile angles and curves, respectively. The use of touch was different – passive touch when receiving the metallic pins and haptic touch when actively exploring the tactile (angle; curve) shapes, and the added auditory sound was different, that is, task-irrelevant (sinusoidal tones) and task-relevant (congruous; incongruous, angular; curved) sounds. Moreover, the participants who were sighted in Hötting and Röder’s (2004) study were asked to fixate their vision (and maintain it) on a visual fixation cross immediately in front of them throughout the experiment. In this study, the participants who had normal/corrected-to-normal vision were blindfolded. So, the participants who were sighted received tactile (task-relevant), auditory (task-irrelevant), and visual (task-irrelevant) information in Hötting and Röder’s (2004) study, and tactile (task-relevant) and auditory (task-relevant) information in this study.
Interestingly, therefore, Graven and Desebrock (2018) have suggested that people integrate tactile and auditory (angle; curve) shape information, regardless of whether they are experienced or inexperienced in using haptic touch (with no visual information: cf. Hötting & Röder, 2004), but those who are experienced integrate it only when they find it necessary. According to this suggestion, the participants who were inexperienced in using haptic touch in this study should integrate the congruous tactile and auditory (angle; curve) shape information. The participants who were experienced in this study should do this if they find it necessary (i.e., that the tactile angles and curves were too difficult to recognise by haptic touch alone. See also the angle-curve mix-up in Graven, 2016 – haptic touch mixes up ≈70% of tactile 2D angles and curves, which makes it difficult for people to recognise even the most common shapes). All of them did this. Thus, this study suggests that the difficulty in recognising the tactile angles and curves, and not the experience in using haptic touch, determines whether or not people integrate the tactile and the auditory (angle; curve) shape information. Further research is needed, therefore, to investigate whether adding auditory angular and curved sounds to tactile angle and curve shapes – one unspecified sound to one unspecified shape – influences what discrimination strategy is adopted by haptic touch, and thus the recognition proficiency (see Graven, 2016).
Graven and Desebrock (2018) have also suggested that people who are experienced in using haptic touch know when to ignore certain sensory information, and they typically ignore the auditory (angular; curved) sounds, but not people who are inexperienced in using it. According to this suggestion, the participants who were experienced in using haptic touch in this study should ignore the auditory (angular; curved) sounds in the incongruous condition, and the participants who were inexperienced should not. Revisiting Figure 3 (incongruous condition), nine of the 13 participants who were experienced (B; D; E; F; G; H; I; J; M) and eight of the 12 who were inexperienced in using haptic touch (A; C; E; G; H; I; K; L) correctly ignored the auditory (angular; curved) sounds (in ⩾4 and ⩾3 of the eight tactile angles and curves, respectively. Four experienced [A; C; K; L] and four inexperienced [B; D; F; J] did not). Indeed, this study somewhat contradicts Graven and Desebrock’s (2018) suggestion. This study suggests that people know when to ignore certain sensory information regardless of whether they are experienced or inexperienced in using haptic touch. Further research is needed, however, to investigate whether it is the difficulty in recognising the tactile (angle; curve) shape information (see the angle-curve mix-up in Graven, 2016), the experience in using haptic touch (see Graven and Desebrock, 2018), that is, amount or type, or all of the above in combination that determines if people know when to ignore the auditory (angle; curve) shape information or not. In respect of the people who do not know when to ignore certain sensory information, further research is needed to investigate whether they become distracted by the auditory (angle; curve) shape information (see, e.g., Jacoby et al., 2012; Lavie, 1995; Lavie et al., 2009; Macdonald & Lavie, 2011; Sandhu & Dyson, 2016; Tellinghuisen et al., 2016; Treisman, 1995; Wolfe & Robertson, 2012) or whether they too know when to ignore it; that is, they ignore the auditory (angular; curved) sounds but fail to recognise the tactile (angle; curve) shapes by haptic touch alone (see the angle-curve mix-up in Graven, 2016).
Adding congruous auditory angular and curved sounds to tactile angle and curve shapes positively influences the proficiency in recognising tactile angles and curves – people integrate tactile and auditory (angle; curve) shape information, and this improves their recognition proficiency. Thus, this study suggests that sound-shape regularities do exist (see also Graven & Desebrock, 2018, 2019). Further research is needed, however, to investigate how these sound-shape regularities can be used to positively influence the proficiency of haptic touch in recognising other tactile shape information, if at all, and how robust they are.
Footnotes
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement number 657440 and by the Norwegian Association of the Blind and Partially Sighted.
