Fluidity in the perception of auditory speech: Cross-modal recalibration of voice gender and vowel identity by a talking face

Abstract

Humans quickly adapt to variations in the speech signal. Adaptation may surface as recalibration, a learning effect driven by error-minimisation between a visual face and an ambiguous auditory speech signal, or as selective adaptation, a contrastive aftereffect driven by the acoustic clarity of the sound. Here, we examined whether these aftereffects occur for vowel identity and voice gender. Participants were exposed to male, female, or androgynous tokens of speakers pronouncing /e/, /ø/, (embedded in words with a consonant-vowel-consonant structure), or an ambiguous vowel halfway between /e/ and /ø/ dubbed onto the video of a male or female speaker pronouncing /e/ or /ø/. For both voice gender and vowel identity, we found assimilative aftereffects after exposure to auditory ambiguous adapter sounds, and contrastive aftereffects after exposure to auditory clear adapter sounds. This demonstrates that similar principles for adaptation in these dimensions are at play.

Keywords

Audiovisual integration gender vowel recalibration selective adaptation

Get full access to this article

View all access options for this article.

References

Arnal

L. H.

Morillon

Kell

C. A.

Giraud

A. L.

(2009). Dual neural routing of visual facilitation in speech processing. Journal of Neuroscience, 29(43), 13445–13453. https://doi.org/10.1523/JNEUROSCI.3194-09.2009

Baart

de Boer-Schellekens

Vroomen

(2012). Lipread-induced phonetic recalibration in dyslexia. Acta Psychologica, 140(1), 91–95. https://doi.org/10.1016/j.actpsy.2012.03.003

Baart

Lindborg

Andersen

T. S.

(2017). Electrophysiological evidence for differences between fusion and combination illusions in audiovisual speech perception. European Journal of Neuroscience, 46(10), 2578–2583. https://doi.org/10.1111/ejn.13734

Baart

Stekelenburg

J. J.

Vroomen

(2014). Electrophysiological evidence for speech-specific audiovisual integration. Neuropsychologia, 53, 115–121. https://doi.org/10.1016/j.neuropsychologia.2013.11.011

Baart

Vroomen

(2010). Phonetic recalibration does not depend on working memory. Experimental Brain Research, 203(3), 575–582. https://doi.org/10.1007/s00221-010-2264-9

Baart

Vroomen

(2018). Recalibration of vocal affect by a dynamic face. Experimental Brain Research, 236, 1911–1918.

Belin

Fecteau

Bedard

(2004). Thinking the voice: Neural correlates of voice perception. Trends in Cognitive Sciences, 8(3), 129–135.

Belin

Zatorre

R. J.

Lafaille

Ahad

Pike

(2000). Voice-selective areas in human auditory cortex. Nature, 403(6767), 309–312.

Bermant

R. I.

Welch

R. B.

(1976). Effect of degree of separation of visual-auditory stimulus and eye position upon spatial interaction of vision and audition. Perceptual and Motor Skills, 43(2), 487–493. https://doi.org/10.2466/pms.1976.43.2.487

10.

Bertelson

Aschersleben

(1998). Automatic visual bias of perceived auditory location. Psychonomic Bulletin & Review, 5(3), 482–489.

11.

Bertelson

Vroomen

de Gelder

(2003). Visual recalibration of auditory speech identification: A McGurk aftereffect. Psychological Science, 14(6), 592–597. https://doi.org/10.1046/j.0956-7976.2003.psci_1470.x

12.

Bestelmeyer

P. E.

Belin

Grosbras

M. H.

(2011). Right temporal TMS impairs voice detection. Current Biology, 21(20), R838–R839. https://doi.org/10.1016/j.cub.

13.

Blank

Anwander

von Kriegstein

(2011). Direct structural connections between voice-and face-recognition areas. Journal of Neuroscience, 31(96), 12906–12915. https://doi.org/10.1523/JNEUROSCI.2091-11.2011

14.

Bonte

Correia

J. M.

Keetels

Vroomen

Formisano

(2017). Reading-induced shifts of perceptual speech representations in auditory cortex. Scientific Reports, 7, 5143. https://doi.org/10.1038/s41598-017-05356-3

15.

Burgering

M. A.

Baart

Vroomen

(2018, June 14–17). Audiovisual recalibration and selective adaptation for vowels and speaker sex [Conference session]. 19th International Multisensory Research Forum (IMRF), Toronto, ON, Canada.

16.

Campanella

Belin

(2007). Integrating face and voice in person perception. Trends in Cognitive Sciences, 11(12), 535–543. https://doi.org/10.1016/j.tics.2007.10.001

17.

Cellerino

Borghetti

Sartucci

(2004). Sex differences in face gender recognition in humans. Brain Research Bulletin, 63(6), 443–449. https://doi.org/10.1016/j.brainresbull.2004.03.010

18.

Charest

Pernet

Latinus

Crabbe

Belin

(2012). Cerebral processing of voice gender studied using a continuous carryover fMRI design. Cerebral Cortex, 23(4), 958–966.

19.

Curby

K. M.

Johnson

K. J.

Tyson

(2012). Face to face with emotion: Holistic face processing is modulated by emotional state. Cognition & Emotion, 26(1), 93–102. https://doi.org/10.1080/02699931.2011.555752

20.

de Gelder

Vroomen

. (2000). The perception of emotions by ear and by eye. Cognition & Emotion, 14(3), 289–311.

21.

Diehl

R. L.

(1975). The effect of selective adaptation on the identification of speech sounds. Perception & Psychophysics, 17(1), 48–52.

22.

Eimas

P. D.

Corbit

J. D.

(1973). Selective adaptation of linguistic feature detectors. Cognitive Psychology, 4(1), 99–109.

23.

Fenn

K. M.

Shintel

Atkins

A. S.

Skipper

J. I.

Bond

V. C.

Nusbaum

H. C.

(2011). When less is heard than meets the ear: Change deafness in a telephone conversation. Quarterly Journal of Experimental Psychology, 64(7), 1442–1456. https://doi.org/10.1080/17470218.2011.570353

24.

Franken

Eisner

Schoffelen

Acheson

D. J.

Hagoort

McQueen

J. M.

(2017). Audiovisual recalibration of vowel categories [Conference session]. Proceedings of Interspeech 2017. https://pure.mpg.de/pubman/faces/ViewItemOverviewPage.jsp?itemId=item_2445791_9

25.

Fujisaki

Shimojo

Kashino

Nishida

S. Y.

(2004). Recalibration of audiovisual simultaneity. Nature Neuroscience, 7(7), 773–778.

26.

Gelfer

M. P.

Mikos

V. A.

(2005). The relative contributions of speaking fundamental frequency and formant frequencies to gender identification based on isolated vowels. Journal of Voice, 19(4), 544–554. https://doi.org/10.1016/j.jvoice.2004.10.006

27.

Green

K. P.

Kuhl

P. K.

Meltzoff

A. N.

Stevens

E. B.

(1991). Integrating speech information across talkers, gender, and sensory modality: Female faces and male voices in the McGurk effect. Perception & Psychophysics, 50(6), 524–536.

28.

Huestegge

S. M.

Raettig

(2018). Crossing gender borders: Bidirectional dynamic interaction between face-based and voice-based gender categorization. Journal of Voice. Advance online publication. https://doi.org/10.1016/j.jvoice.2018.09.020

29.

Imaizumi

Mori

Kiritani

Kawashima

Sugiura

Fukuda

Hatano

(1997). Vocal identification of speaker and emotion activates different brain regions. NeuroReport, 8(12), 2809–2812.

30.

Joassin

Maurage

Campanella

(2011). The neural network sustaining the crossmodal processing of human gender from faces and voices: An fMRI study. Neuroimage, 54(2), 1654–1661.

31.

Johnson

Strand

E. A.

D’Imperio

(1999). Auditory–visual integration of talker gender in vowel perception. Journal of Phonetics, 27(4), 359–384.

32.

Kawahara

Morise

Takahashi

Nisimura

Irino

Banno

(2008). Tandem-STRAIGHT: A temporally stable power spectral representation for periodic signals and applications to interference-free spectrum, F0, and aperiodicity estimation. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2008, pp. 3933–3936). Institute of Electrical and Electronics Engineers.

33.

Keetels

Bonte

Vroomen

(2018). A selective deficit in phonetic recalibration by text in developmental dyslexia. Frontiers in Psychology, 9, 710.

34.

Keetels

Pecoraro

Vroomen

(2015). Recalibration of auditory phonemes by lipread speech is ear-specific. Cognition, 141, 121–126. https://doi.org/10.1016/j.cognition.2015.04.019

35.

Keetels

Stekelenburg

J. J.

Vroomen

(2016). A spatial gradient in phonetic recalibration by lipread speech. Journal of Phonetics, 56, 124–130. https://doi.org/10.1016/j.wocn.2016.02.005

36.

Keetels

Vroomen

(2007). No effect of auditory-visual spatial disparity on temporal recalibration. Experimental Brain Research, 182(4), 559–565.

37.

Kilian-Hütten

Valente

Vroomen

Formisano

(2011). Auditory cortex encodes the perceptual interpretation of ambiguous sound. Journal of Neuroscience, 31(5), 1715–1720.

38.

Kilian-Hütten

Vroomen

Formisano

(2011). Brain activation during audiovisual exposure anticipates future perception of ambiguous speech. Neuroimage, 57(4), 1601–1607. https://doi.org/10.1016/j.neuroimage.2011.05.043

39.

Kleinschmidt

Jaeger

T. F.

(2011). A Bayesian belief updating model of phonetic recalibration and selective adaptation. In Proceedings of the 2nd Workshop on Cognitive Modeling and Computational Linguistics (pp. 10–19). Association for Computational Linguistics.

40.

Klucharev

Möttönen

Sams

(2003). Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception. Cognitive Brain Research, 18(1), 65–75. https://doi.org/10.1016/j.cogbrainres.2003.09.004

41.

Kuhl

P. K.

(1991). Human adults and human infants show a “perceptual magnet effect” for the prototypes of speech categories, monkeys do not. Perception & Psychophysics, 50(2), 93–107.

42.

Latinus

Taylor

M. J.

(2012). Discriminating male and female voices: Differentiating pitch and gender. Brain Topography, 25(2), 194–204.

43.

Latinus

VanRullen

Taylor

M. J.

(2010). Top-down and bottom-up modulation in processing bimodal face/voice stimuli. BMC Neuroscience, 11(1), 36.

44.

Lewin

Herlitz

(2002). Sex differences in face recognition—Women’s faces make the difference. Brain and Cognition, 50(1), 121–128.

45.

Pernet

C. R.

Belin

(2012). The role of pitch and timbre in voice gender categorization. Frontiers in Psychology, 3, 23.

46.

Pilling

(2009). Auditory event-related potentials (ERPs) in audiovisual speech perception. Journal of Speech, Language, and Hearing Research, 52(4), 1073–1081.

47.

Radeau

Bertelson

(1974). The after-effects of ventriloquism. Quarterly Journal of Experimental Psychology, 26(1), 63–71.

48.

Roberts

Summerfield

(1981). Audiovisual presentation demonstrates that selective adaptation in speech perception is purely auditory. Perception & Psychophysics, 30(4), 309–314.

49.

Saint-Amour

De Sanctis

Molholm

Ritter

Foxe

J. J.

(2007). Seeing voices: High-density electrical mapping and source-analysis of the multisensory mismatch negativity evoked during the McGurk illusion. Neuropsychologia, 45(3), 587–597. https://doi.org/10.1016/j.neuropsychologia.2006.03.036

50.

Saldaña

H. M.

Rosenblum

L. D.

(1994). Selective adaptation in speech perception using a compelling audiovisual adaptor. The Journal of the Acoustical Society of America, 95(6), 3658–3661.

51.

Schweinberger

S. R.

Casper

Hauthal

Kaufmann

J. M.

Kawahara

Kloth

Zäske

(2008). Auditory adaptation in voice perception. Current Biology, 18, 684–688. https://doi.org/10.1016/j.cub.2008.04.015

52.

Schweinberger

S. R.

Kawahara

Simpson

A. P.

Skuk

V. G.

Zäske

(2014). Speaker perception. Wiley Interdisciplinary Reviews: Cognitive Science, 5(1), 15–25.

53.

Stekelenburg

J. J.

Vroomen

(2007). Neural correlates of multisensory integration of ecologically valid audiovisual events. Journal of Cognitive Neuroscience, 19(12), 1964–1973.

54.

Sugano

Keetels

Vroomen

(2016). Auditory dominance in motor-sensory temporal recalibration. Experimental Brain Research, 234(5), 1249–1262. https://doi.org/10.1007/s00221-015-4497-0

55.

Sumby

W. H.

Pollack

(1954). Visual contribution to speech intelligibility in noise. The Journal of the Acoustical Society of America, 26(2), 212–215.

56.

Sun

Gao

Han

(2010). Sex differences in face gender recognition: An event-related potential study. Brain Research, 1327, 69–76.

57.

Titze

I. R.

(1989). Physiologic and acoustic differences between male and female voices. The Journal of the Acoustical Society of America, 85(4), 1699–1707.

58.

Tottenham

Tanaka

J. W.

Leon

A. C.

McCarry

Nurse

Hare

T. A.

Nelson

(2009). The NimStim set of facial expressions: Judgments from untrained research participants. Psychiatry Research, 168(3), 242–249.

59.

van Linden

Vroomen

. (2007). Recalibration of phonetic categories by lipread speech versus lexical information. Journal of Experimental Psychology: Human Perception & Performance, 33(6), 1483–1494. https://doi.org/10.1037/0096-1523.33.6.1483

60.

van Wassenhove

Grant

K. W.

Poeppel

. (2005). Visual speech speeds up the neural processing of auditory speech. Proceedings of the National Academy of Sciences of the United States of America, 102(4), 1181–1186. https://doi.org/10.1073/pnas.0408949102

61.

Von Kriegstein

Eger

Kleinschmidt

Giraud

A. L

. (2003). Modulation of neural responses to speech by directing attention to voices or verbal content. Cognitive Brain Research, 17(1), 48–55.

62.

Von Kriegstein

Kleinschmidt

Sterzer

Giraud

A. L

. (2005). Interaction of face and voice areas during speaker recognition. Journal of Cognitive Neuroscience, 17(3), 367–376.

63.

von Kriegstein

Smith

D. R. R.

Patterson

R. D.

Kiebel

S. J.

Griffiths

T. D

. (2010). How the human brain recognizes speech in the context of changing speakers. Journal of Neuroscience, 30(2), 629–638.

64.

Vroomen

Baart

(2009). Phonetic recalibration only occurs in speech mode. Cognition, 110(2), 254–259. https://doi.org/10.1016/j.cognition.2008.10.015

65.

Vroomen

Baart

(2012). Phonetic recalibration in audiovisual speech. In Murray

M. M.

Wallace

M. T.

(Eds.), The neural bases of multisensory processes (363–379). CRC Press/Taylor & Francis.

66.

Vroomen

Keetels

De Gelder

Bertelson

(2004). Recalibration of temporal order perception by exposure to audio-visual asynchrony. Cognitive Brain Research, 22(1), 32–35.

67.

Vroomen

van Linden

Keetels

de Gelder

Bertelson

(2004). Selective adaptation and recalibration of auditory speech by lipread information: Dissipation. Speech Communication, 44, 55–61.

68.

Wozny

D. R.

Shams

(2011). Recalibration of auditory space following milliseconds of cross-modal discrepancy. Journal of Neuroscience, 31(12), 4607–4612.

69.

Zäske

Perlich

M. C.

Schweinberger

S. R.

(2016). To hear or not to hear: Voice processing under visual load. Attention Perception & Psychophysics, 78(5), 1488–1495. https://doi.org/10.3758/s13414-016-1119-2

70.

Zäske

Schweinberger

S. R.

Kaufmann

J. M.

Kawahara

(2009). In the ear of the beholder: Neural correlates of adaptation to voice gender. European Journal of Neuroscience, 30, 527–534. https://doi.org/10.1111/j.1460-9568.2009/06839.x

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

8.90 MB