Two experiments investigate the effectiveness of audiovisual (AV) speech cues (cues derived from both seeing and hearing a talker speak) in facilitating perceptual learning of spectrally distorted speech. Speech was distorted through an eight channel noise-vocoder which shifted the spectral envelope of the speech signal to simulate the properties of a cochlear implant with a 6 mm place mismatch. Experiment 1 found that participants showed significantly greater improvement in perceiving noise-vocoded speech when training gave AV cues than when it gave auditory cues alone. Experiment 2 compared training with AV cues with training which gave written feedback. These two methods did not significantly differ in the pattern of training they produced. Suggestions are made about the types of circumstances in which the two training methods might be found to differ in facilitating auditory perceptual learning of speech.
BenchJ.KowalA.BamfordJ. (1979). The BKB (Bamford-Kowal-Bench) sentence lists for partially-hearing children. British Journal of Audiology, 13, 108–112.
2.
BergesonT. R.PisoniD. B.DavisR. A. O. (2005). Development of audiovisual comprehension skills in prelingually deaf children with cochlear implants. Ear & Hearing, 26, 149–164.
3.
DalyN.BenchJ.ChappellH. (1996). Gender differences in speechreadability. Journal of the Academy of Rehabilitative Audiology, 29, 27–40.
4.
DavisC.KimJ. (2004). Audio-visual interactions with intact clearly audible speech. Quarterly Journal of Experimental Psychology A, 57, 1103–1121.
5.
DavisM. H.JohnsrudeI. S.Hervais-AdlemanA.TaylorK.McGettiganC. (2005). Lexical information drives perceptual learning of distorted speech: Evidence from comprehension of noise-vocoded sentences. Journal of Experimental Psychology: General, 34, 222–241.
6.
DemorestM. E.BernsteinL. E. (1992). Sources of variability in speechreading sentences: A generalizability analysis. Journal of Speech & Hearing Research, 35, 876–891.
7.
DohenM.LœvenbruckH.CathiardM. A.SchwartzJ. L. (2004). Visual perception of contrastive focus in reiterant French speech. Speech Communication, 44, 155–172.
8.
DormanM. F.LoizuP. C.RaineyD. (1997). Simulating the effect of cochlear-implant electrode insertion depth on speech understanding. Journal of the Acoustical Society of America, 102, 2993–2996.
9.
DormanM. F.LoizuP. C.FitzkeJ.TuZ. (1998). The recognition of sentences in noise by normal-hearing listeners using simulations of cochlear-implant signal processors with 6–20 channels. Journal of the Acoustical Society of America, 104, 3583–3585.
10.
ErberN. P. (1975). Auditory-visual perception of speech. Journal of Speech and Hearing Disorders, 40, 481–492.
11.
FuQ. J.GalvinJ. J. (2003). The effects of short-term training for spectrally mismatched noise-band speech. Journal of the Acoustical Society of America, 113, 1065–1072.
12.
FuQ. J.NogakiG.GalvinJ. J. (2006). Auditory training with spectrally shifted speech: Implications for cochlear implant patient auditory rehabilitation. Journal of the Association for Research in Otolaryngology, 6, 180–189.
13.
GagnéJ. P.MastersonV. M.MunhallK. G.BilidaN.QuerengesserC. (1994). Across talker variability in auditory, visual, and audiovisual speech intelligibility for conversational and clear speech. Journal of the Academy of Rehabilitative Audiology, 27, 135–158.
14.
GrantK. W.WaldenB. E.SeitzP. F. (1998). Auditory-visual speech recognition by hearing-impaired subjects: Consonant recognition, sentence recognition, and auditory-visual integration. Journal of the Acoustical Society of America, 103, 2677–2690.
15.
GreenK. P. (1997). The use of auditory and visual information during phonetic processing: Implications for theories of speech perception. In CampbellR.DoddB.BurnhamD. (Eds.), Hearing by eye II: Advances in the psychology of speechreading and auditory-visual speech (pp. 3–25). Hove: Psychology Press.
16.
HardisonD. (2003). Acquisition of second-language speech: Effects of visual cues, context and talker variability. Applied Psycholinguist, 24, 495–522.
17.
HazanV.SennemaA.IbaM.FaulknerA. (2005). Effect of audiovisual perceptual training on the perception and production of consonants by Japanese learners of English. Speech Communication, 47, 360–378.
18.
HazenT. J. (2006). Visual model structures and synchrony constraints for audio-visual speech recognition. IEEE Transactions on Audio, Speech, and Language Processing, 14, 1082–1089.
19.
HelferK. S.FreymanR. L. (2005). The role of visual speech cues in reducing energetic and informational masking. Journal of the Acoustical Society of America, 117, 842–849.
20.
Hervais-AdelmanA.DavisM. H.JohnsrudeI. S.CarlyonR. P. (2008). Perceptual learning of noise-vocoded words: Effects of feedback and lexicality. Journal of Experimental Psychology: Human Perception & Performance, 34, 460–474.
21.
JamesD.BrintonJ.RajputK.GoswamiU. (2008). Phonological awareness, vocabulary, and word reading in children who use cochlear implants: Does age of implantation explain individual variability in performance outcomes and growth?Journal of Deaf Studies and Deaf Education, 13, 117–137.
22.
KricosP. B.LesnerS. A. (1982). Differences in visual intelligibility across talkers. The Volta Review, 84, 219–225.
23.
LesnerS. A. (1988). The talker. The Volta Review, 90, 89–98.
24.
MacLeodA.SummerfieldQ. (1987). Quantifying the contribution of vision to speech perception in noise. British Journal of Audiology, 12, 131–141.
RosenS.FaulknerA.WilkinsonL. (1999). Perceptual adaptation by normal listeners to upwards shifts of spectral information in speech and its relevance for users of cochlear implants. Journal of the Acoustical Society of America, 106, 3629–3636.
30.
RotaG.TuricchiaL.VeitR.GuazzelliM.BirbaumerN.DogilfG. (2008). Perceptual learning of speech processed by a Cochlear Implant simulator: An fMRI investigation. International Journal of Psychophysiology, 69, 225–226.
31.
SaldañaH. M.PisoniD. B.FellowesJ. M.RemezR. E. (1996). Audio-visual speech perception without speech cues. Proceedings of the 4th International Conference on Spoken Language Processing, Philadelphia, PA, 2187–2190.
32.
SchwartzJ. L.BerthommierF.SavariauxC. (2004). Seeing to hear better: Evidence for early audio-visual interactions in speech identification. Cognition, 93, B69–B78.
ShannonR. V.ZengF. G.WygonskiJ. (1998). Speech recognition with altered spectral distribution of envelope cues. Journal of the Acoustical Society of America, 104, 2467–2476.
35.
Soto-FaracoS.NavarraJ.AlsiusA. (2004). Assessing automaticity in audiovisual speech integration: Evidence from the speeded classification task. Cognition, 92, B13–B23.
36.
StaceyP.SummerfieldQ. (2007). Effectiveness of computer-based auditory training in improving the perception of noise-vocoded speech. Journal of the Acoustical Society of America, 121, 2923–2935.
37.
StaceyP.SummerfieldQ. (2008). Comparison of word-, sentence-, and phoneme-based training strategies in improving the perception of spectrally distorted speech. Journal of Speech, Language & Hearing Research, 51, 526–538.
38.
SummerfieldQ. (1987). Some preliminaries to a comprehensive account of audio-visual speech perception. In DoddB.CampbellR. (Eds.), Hearing by eye: The psychology of lip-reading (pp. 3–51). London: Lawrence Erlbaum.
39.
TeinonenT.AslinR. N.AlkuP.CsibraG. (2008). Visual speech contributes to phonetic learning in 6-month-old infants. Cognition, 3, 850–855.
40.
van WassenhoveV.GrantK. W.PoeppelD. (2005). Visual speech speeds up the neural processing of auditory speech. Proceedings of the National Academy of Science, 102, 1181–1186.
