Humans quickly adapt to variations in the speech signal. Adaptation may surface as recalibration, a learning effect driven by error-minimisation between a visual face and an ambiguous auditory speech signal, or as selective adaptation, a contrastive aftereffect driven by the acoustic clarity of the sound. Here, we examined whether these aftereffects occur for vowel identity and voice gender. Participants were exposed to male, female, or androgynous tokens of speakers pronouncing /e/, /ø/, (embedded in words with a consonant-vowel-consonant structure), or an ambiguous vowel halfway between /e/ and /ø/ dubbed onto the video of a male or female speaker pronouncing /e/ or /ø/. For both voice gender and vowel identity, we found assimilative aftereffects after exposure to auditory ambiguous adapter sounds, and contrastive aftereffects after exposure to auditory clear adapter sounds. This demonstrates that similar principles for adaptation in these dimensions are at play.
ArnalL. H.MorillonB.KellC. A.GiraudA. L. (2009). Dual neural routing of visual facilitation in speech processing. Journal of Neuroscience, 29(43), 13445–13453. https://doi.org/10.1523/JNEUROSCI.3194-09.2009
BaartM.LindborgA.AndersenT. S. (2017). Electrophysiological evidence for differences between fusion and combination illusions in audiovisual speech perception. European Journal of Neuroscience, 46(10), 2578–2583. https://doi.org/10.1111/ejn.13734
BaartM.VroomenJ. (2010). Phonetic recalibration does not depend on working memory. Experimental Brain Research, 203(3), 575–582. https://doi.org/10.1007/s00221-010-2264-9
6.
BaartM.VroomenJ. (2018). Recalibration of vocal affect by a dynamic face. Experimental Brain Research, 236, 1911–1918.
7.
BelinP.FecteauS.BedardC. (2004). Thinking the voice: Neural correlates of voice perception. Trends in Cognitive Sciences, 8(3), 129–135.
8.
BelinP.ZatorreR. J.LafailleP.AhadP.PikeB. (2000). Voice-selective areas in human auditory cortex. Nature, 403(6767), 309–312.
9.
BermantR. I.WelchR. B. (1976). Effect of degree of separation of visual-auditory stimulus and eye position upon spatial interaction of vision and audition. Perceptual and Motor Skills, 43(2), 487–493. https://doi.org/10.2466/pms.1976.43.2.487
BertelsonP.VroomenJ.de GelderB. (2003). Visual recalibration of auditory speech identification: A McGurk aftereffect. Psychological Science, 14(6), 592–597. https://doi.org/10.1046/j.0956-7976.2003.psci_1470.x
12.
BestelmeyerP. E.BelinP.GrosbrasM. H. (2011). Right temporal TMS impairs voice detection. Current Biology, 21(20), R838–R839. https://doi.org/10.1016/j.cub.
13.
BlankH.AnwanderA.von KriegsteinK. (2011). Direct structural connections between voice-and face-recognition areas. Journal of Neuroscience, 31(96), 12906–12915. https://doi.org/10.1523/JNEUROSCI.2091-11.2011
14.
BonteM.CorreiaJ. M.KeetelsM.VroomenJ.FormisanoE. (2017). Reading-induced shifts of perceptual speech representations in auditory cortex. Scientific Reports, 7, 5143. https://doi.org/10.1038/s41598-017-05356-3
15.
BurgeringM. A.BaartM.VroomenJ. (2018, June14–17). Audiovisual recalibration and selective adaptation for vowels and speaker sex [Conference session]. 19th International Multisensory Research Forum (IMRF), Toronto, ON, Canada.
CharestI.PernetC.LatinusM.CrabbeF.BelinP. (2012). Cerebral processing of voice gender studied using a continuous carryover fMRI design. Cerebral Cortex, 23(4), 958–966.
19.
CurbyK. M.JohnsonK. J.TysonA. (2012). Face to face with emotion: Holistic face processing is modulated by emotional state. Cognition & Emotion, 26(1), 93–102. https://doi.org/10.1080/02699931.2011.555752
20.
de GelderB.VroomenJ. (2000). The perception of emotions by ear and by eye. Cognition & Emotion, 14(3), 289–311.
21.
DiehlR. L. (1975). The effect of selective adaptation on the identification of speech sounds. Perception & Psychophysics, 17(1), 48–52.
22.
EimasP. D.CorbitJ. D. (1973). Selective adaptation of linguistic feature detectors. Cognitive Psychology, 4(1), 99–109.
23.
FennK. M.ShintelH.AtkinsA. S.SkipperJ. I.BondV. C.NusbaumH. C. (2011). When less is heard than meets the ear: Change deafness in a telephone conversation. Quarterly Journal of Experimental Psychology, 64(7), 1442–1456. https://doi.org/10.1080/17470218.2011.570353
FujisakiW.ShimojoS.KashinoM.NishidaS. Y. (2004). Recalibration of audiovisual simultaneity. Nature Neuroscience, 7(7), 773–778.
26.
GelferM. P.MikosV. A. (2005). The relative contributions of speaking fundamental frequency and formant frequencies to gender identification based on isolated vowels. Journal of Voice, 19(4), 544–554. https://doi.org/10.1016/j.jvoice.2004.10.006
27.
GreenK. P.KuhlP. K.MeltzoffA. N.StevensE. B. (1991). Integrating speech information across talkers, gender, and sensory modality: Female faces and male voices in the McGurk effect. Perception & Psychophysics, 50(6), 524–536.
28.
HuesteggeS. M.RaettigT. (2018). Crossing gender borders: Bidirectional dynamic interaction between face-based and voice-based gender categorization. Journal of Voice. Advance online publication. https://doi.org/10.1016/j.jvoice.2018.09.020
29.
ImaizumiS.MoriK.KiritaniS.KawashimaR.SugiuraM.FukudaH.HatanoK. (1997). Vocal identification of speaker and emotion activates different brain regions. NeuroReport, 8(12), 2809–2812.
30.
JoassinF.MaurageP.CampanellaS. (2011). The neural network sustaining the crossmodal processing of human gender from faces and voices: An fMRI study. Neuroimage, 54(2), 1654–1661.
31.
JohnsonK.StrandE. A.D’ImperioM. (1999). Auditory–visual integration of talker gender in vowel perception. Journal of Phonetics, 27(4), 359–384.
32.
KawaharaH.MoriseM.TakahashiT.NisimuraR.IrinoT.BannoH. (2008). Tandem-STRAIGHT: A temporally stable power spectral representation for periodic signals and applications to interference-free spectrum, F0, and aperiodicity estimation. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2008, pp. 3933–3936). Institute of Electrical and Electronics Engineers.
33.
KeetelsM.BonteM.VroomenJ. (2018). A selective deficit in phonetic recalibration by text in developmental dyslexia. Frontiers in Psychology, 9, 710.
KeetelsM.StekelenburgJ. J.VroomenJ. (2016). A spatial gradient in phonetic recalibration by lipread speech. Journal of Phonetics, 56, 124–130. https://doi.org/10.1016/j.wocn.2016.02.005
36.
KeetelsM.VroomenJ. (2007). No effect of auditory-visual spatial disparity on temporal recalibration. Experimental Brain Research, 182(4), 559–565.
37.
Kilian-HüttenN.ValenteG.VroomenJ.FormisanoE. (2011). Auditory cortex encodes the perceptual interpretation of ambiguous sound. Journal of Neuroscience, 31(5), 1715–1720.
38.
Kilian-HüttenN.VroomenJ.FormisanoE. (2011). Brain activation during audiovisual exposure anticipates future perception of ambiguous speech. Neuroimage, 57(4), 1601–1607. https://doi.org/10.1016/j.neuroimage.2011.05.043
39.
KleinschmidtD.JaegerT. F. (2011). A Bayesian belief updating model of phonetic recalibration and selective adaptation. In Proceedings of the 2nd Workshop on Cognitive Modeling and Computational Linguistics (pp. 10–19). Association for Computational Linguistics.
40.
KlucharevV.MöttönenR.SamsM. (2003). Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception. Cognitive Brain Research, 18(1), 65–75. https://doi.org/10.1016/j.cogbrainres.2003.09.004
41.
KuhlP. K. (1991). Human adults and human infants show a “perceptual magnet effect” for the prototypes of speech categories, monkeys do not. Perception & Psychophysics, 50(2), 93–107.
42.
LatinusM.TaylorM. J. (2012). Discriminating male and female voices: Differentiating pitch and gender. Brain Topography, 25(2), 194–204.
43.
LatinusM.VanRullenR.TaylorM. J. (2010). Top-down and bottom-up modulation in processing bimodal face/voice stimuli. BMC Neuroscience, 11(1), 36.
44.
LewinC.HerlitzA. (2002). Sex differences in face recognition—Women’s faces make the difference. Brain and Cognition, 50(1), 121–128.
45.
PernetC. R.BelinP. (2012). The role of pitch and timbre in voice gender categorization. Frontiers in Psychology, 3, 23.
46.
PillingM. (2009). Auditory event-related potentials (ERPs) in audiovisual speech perception. Journal of Speech, Language, and Hearing Research, 52(4), 1073–1081.
47.
RadeauM.BertelsonP. (1974). The after-effects of ventriloquism. Quarterly Journal of Experimental Psychology, 26(1), 63–71.
48.
RobertsM.SummerfieldQ. (1981). Audiovisual presentation demonstrates that selective adaptation in speech perception is purely auditory. Perception & Psychophysics, 30(4), 309–314.
49.
Saint-AmourD.De SanctisP.MolholmS.RitterW.FoxeJ. J. (2007). Seeing voices: High-density electrical mapping and source-analysis of the multisensory mismatch negativity evoked during the McGurk illusion. Neuropsychologia, 45(3), 587–597. https://doi.org/10.1016/j.neuropsychologia.2006.03.036
50.
SaldañaH. M.RosenblumL. D. (1994). Selective adaptation in speech perception using a compelling audiovisual adaptor. The Journal of the Acoustical Society of America, 95(6), 3658–3661.
51.
SchweinbergerS. R.CasperC.HauthalN.KaufmannJ. M.KawaharaH.KlothN.ZäskeR. (2008). Auditory adaptation in voice perception. Current Biology, 18, 684–688. https://doi.org/10.1016/j.cub.2008.04.015
SumbyW. H.PollackI. (1954). Visual contribution to speech intelligibility in noise. The Journal of the Acoustical Society of America, 26(2), 212–215.
56.
SunY.GaoX.HanS. (2010). Sex differences in face gender recognition: An event-related potential study. Brain Research, 1327, 69–76.
57.
TitzeI. R. (1989). Physiologic and acoustic differences between male and female voices. The Journal of the Acoustical Society of America, 85(4), 1699–1707.
58.
TottenhamN.TanakaJ. W.LeonA. C.McCarryT.NurseM.HareT. A.NelsonC. (2009). The NimStim set of facial expressions: Judgments from untrained research participants. Psychiatry Research, 168(3), 242–249.
59.
van LindenS.VroomenJ. (2007). Recalibration of phonetic categories by lipread speech versus lexical information. Journal of Experimental Psychology: Human Perception & Performance, 33(6), 1483–1494. https://doi.org/10.1037/0096-1523.33.6.1483
60.
van WassenhoveV.GrantK. W.PoeppelD. (2005). Visual speech speeds up the neural processing of auditory speech. Proceedings of the National Academy of Sciences of the United States of America, 102(4), 1181–1186. https://doi.org/10.1073/pnas.0408949102
61.
Von KriegsteinK.EgerE.KleinschmidtA.GiraudA. L. (2003). Modulation of neural responses to speech by directing attention to voices or verbal content. Cognitive Brain Research, 17(1), 48–55.
62.
Von KriegsteinK.KleinschmidtA.SterzerP.GiraudA. L. (2005). Interaction of face and voice areas during speaker recognition. Journal of Cognitive Neuroscience, 17(3), 367–376.
63.
von KriegsteinK.SmithD. R. R.PattersonR. D.KiebelS. J.GriffithsT. D. (2010). How the human brain recognizes speech in the context of changing speakers. Journal of Neuroscience, 30(2), 629–638.
VroomenJ.BaartM. (2012). Phonetic recalibration in audiovisual speech. In MurrayM. M.WallaceM. T. (Eds.), The neural bases of multisensory processes (363–379). CRC Press/Taylor & Francis.
66.
VroomenJ.KeetelsM.De GelderB.BertelsonP. (2004). Recalibration of temporal order perception by exposure to audio-visual asynchrony. Cognitive Brain Research, 22(1), 32–35.
67.
VroomenJ.van LindenS.KeetelsM.de GelderB.BertelsonP. (2004). Selective adaptation and recalibration of auditory speech by lipread information: Dissipation. Speech Communication, 44, 55–61.
68.
WoznyD. R.ShamsL. (2011). Recalibration of auditory space following milliseconds of cross-modal discrepancy. Journal of Neuroscience, 31(12), 4607–4612.
69.
ZäskeR.PerlichM. C.SchweinbergerS. R. (2016). To hear or not to hear: Voice processing under visual load. Attention Perception & Psychophysics, 78(5), 1488–1495. https://doi.org/10.3758/s13414-016-1119-2
70.
ZäskeR.SchweinbergerS. R.KaufmannJ. M.KawaharaH. (2009). In the ear of the beholder: Neural correlates of adaptation to voice gender. European Journal of Neuroscience, 30, 527–534. https://doi.org/10.1111/j.1460-9568.2009/06839.x
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
