Processing of incongruent emotional expressions in voice and semantics: The dominant modality and integration with facial expressions

Abstract

This research concerns three channels for emotional communication: voice, semantics, and facial expressions. We used speech in which the emotion in voice and semantics did not match, and we investigated the dominant modality and how they interact with facial expressions. The study used voices emoting anger, happiness, or sadness while saying, “I’m angry,” “I’m pleased,” or “I’m sad.” A facial image accompanied the voice, and it expressed either the same emotion to the voice (voice = face condition), the same emotion to the semantics (semantic = face condition), or a mixed emotion shown in the voice and semantics (morph condition). The phrases were articulated in the participants’ native language (Japanese), second language (English), and unfamiliar language (Khmer). In Study 1, participants answered how much they agreed that the speaker expressed anger, happiness, and sadness. Their attention was not controlled. In Study 2, participants were told to attend to either voice or semantics. The morph condition of study 1 found semantic dominance for the native language stimuli. The semantic = face and voice = face conditions in Studies 1 and 2 revealed that an emotion solely expressed in semantics (while a different emotion was shown in face and voice) had more substantial impacts on assessing the speaker’s emotion than an emotion solely expressed in voice when the semantics were in understandable languages.

Keywords

Emotion voice semantic content facial expressions incongruent speech

Get full access to this article

View all access options for this article.

References

Astésano

Besson

Alter

(2004). Brain potentials during semantic and prosodic processing in French. Cognitive Brain Research, 18, 172–184. https://doi.org/10.1016/j.cogbrainres.2003.10.002

Barrett

L. F.

(2006). Solving the emotion paradox: Categorization and the experience of emotion. Personality and Social Psychology Review, 10(1), 20–46. https://doi.org/10.1207/s15327957pspr1001_2

Ben-David

B. M.

Multani

Shakuf

Rudzicz

van Lieshout

P. H.

(2016). Prosody and semantics are separate but not separable channels in the perception of emotional speech: Test for Rating of Emotions in Speech. Journal of Speech, Language, and Hearing Research, 59, 72–79. https://doi.org/10.1044/2015_JSLHR-H-14-0323

Collignon

Girard

Gosselin

Roy

Saint-Amour

Lassonde

Lepore

(2008). Audio-visual integration of emotion expression. Brain Research, 1242, 126–135. https://doi.org/10.1016/j.brainres.2008.04.023

de Gelder

Vroomen

. (2000). The perception of emotions by ear and by eye. Cognition and Emotion, 14(3), 289–311. https://doi.org/10.1080/026999300378824

Doyle

C. M.

Lindquist

K. A.

(2017). Language and emotion: Hypotheses on the constructed nature of emotion perception. In Russell

J. A.

Dols Fernandez

J. M.

(Eds.), The science of facial expression (pp. 415–432). Oxford University Press. https://doi.org/10.1093/acprof:oso/9780190613501.003.0022

Ekman

(1992). An argument for basic emotions. Cognition and Emotion, 6, 169–200. https://doi.org/10.1080/02699939208411068

Filippi

Ocklenburg

Bowling

D. L.

Heege

Güntürkün

Newen

de Boer

(2017). More than words (and faces): Evidence for a Stroop effect of prosody in emotion word processing. Cognition and Emotion, 31(5), 879–891. https://doi.org/10.1080/02699931.2016.1177489

Föcker

Gondan

Röder

(2011). Preattentive processing of audio-visual emotional signals. Acta Psychologica, 137(1), 36–47. https://doi.org/10.1016/j.actpsy.2011.02.004

10.

Hawk

S. T.

van Kleef

G. A.

Fischer

A. H.

van der Schalk

(2009). “Worth a thousand words”: Absolute and relative decoding of nonlinguistic affect vocalizations. Emotion, 9(3), 293–305. https://doi.org/10.1037/a0015178

11.

Ishii

Reyes

J. A.

Kitayama

(2003). Spontaneous attention to word content versus emotional tone: Differences among three cultures. Psychological Science, 14, 39–46. https://doi.org/10.1111/1467-9280.01416

12.

Jacob

Brück

Plewnia

Wildgruber

(2014). Cerebral processing of prosodic emotional signals: Evaluation of a network model using rTMS. PLoS ONE, 9, e105509. https://doi.org/10.1371/journal.pone.0105509

13.

Kikutani

Ikemoto

(2022). Detecting emotion in speech expressing incongruent emotional cues through voice and content: investigation on dominant modality and language. Cognition and Emotion, 36(3), 492–511. https://doi.org/10.1080/02699931.2021.2021144

14.

Kitayama

Ishii

(2002). Word and voice: Spontaneous attention to emotional utterances in two languages. Cognition and Emotion, 16, 29–59. https://doi.org/10.1080/0269993943000121

15.

Kim

Davis

(2012). Perceiving emotion from a talker: How face and voice work together. Visual Cognition, 20(8), 902–921. https://doi.org/10.1080/13506285.2012.713874

16.

Kotz

S. A.

Paulmann

(2007). When emotional prosody and semantics dance cheek to cheek: ERP evidence. Brain Research, 115, 107–118. https://doi.org/10.1016/j.brainres.2007.03.015

17.

Langner

Dotsch

Bijlstra

Wigboldus

D. H. J.

Hawk

S. T.

van Knippenberg

(2010). Presentation and validation of the Radboud Faces Database. Cognition and Emotion, 24(8), 1377–1388. https://doi.org/10.1080/02699930903485076

18.

Lindquist

K. A.

MacCormack

J. K.

Shablack

(2015). The role of language in emotion: predictions from psychological constructionism. Frontiers in Psychology, 6, 444. https://doi.org/10.3389/fpsyg.2015.00444

19.

Mehrabian

(1981). Silent messages: Implicit communication of emotions and attitudes. (2nd ed.). Wadsworth.

20.

Nygaard

L. C.

Lunders

E. R.

(2002). Resolution of lexical ambiguity by emotional tone of voice. Memory and Cognition, 30, 583–599. https://doi.org/10.3758/BF03194959

21.

Öhman

Lundqvist

Esteves

(2001). The face in the crowd revisited: A threat advantage with schematic stimuli. Journal of Personality and Social Psychology, 80, 381–396. https://doi.org/10.1037/0022-3514.80.3.381

22.

Ogawa

Oda

Yoshikawa

Akamatsu

(1997). Evaluation of facial expressions differing in face angles—Constructing a database of facial expressions [In Japanese with English abstract]. Technical Report of Institute of Electronics, Information and Communication Engineers (IEICE), HIP, 97(136), 47–52.

23.

Paulmann

Kotz

S. A.

(2008). An ERP investigation on the temporal dynamics of emotional prosody and emotional semantics in pseudo- and lexical-sentence context. Brain and Language, 105, 59–69. https://doi.org/10.1016/j.bandl.2007.11.005

24.

Pell

M. D.

Monetta

Paulmann

Kotz

S. A.

(2009). Recognizing emotions in a foreign language. Journal of Nonverbal Behavior, 33, 107–120. https://doi.org/10.1007/s10919-008-0065-7

25.

Pell

M. D.

Paulmann

Dara

Alasseri

Kotz

S. A.

(2009). Factors in the recognition of vocally expressed emotions: A comparison of four languages. Journal of Phonetics, 37, 417–435. https://doi.org/10.1016/j.wocn.2009.07.005

26.

Pourtois

Dhar

(2013). Integration of face and voice during emotion perception: Is there anything gained for the perceptual system beyond stimulus modality redundancy? In Belin

Campanella

Ethofer

(Eds.), Integrating face and voice in person perception (pp. 181–206). Springer, New York. https://doi.org/10.1007/978-1-4614-3585-3_10

27.

Regenbogen

Schneider

D. A.

Finkelmeyer

Kohn

Derntl

Kellermann

Gur

R. E.

Sxhneider

Habel

(2012). The differential contribution of facial expressions, prosody, and speech content to empathy. Cognition and Emotion, 26(6), 995–1014. https://doi.org/10.1080/02699931.2011.631296

28.

Regenbogen

Schneider

D. A.

Gur

R. E.

Schneider

Habel

Kellermann

(2012). Multimodal human communication—Targeting facial expressions, speech content and prosody. NeuroImage, 60(4), 2346–2356. https://doi.org/10.1016/j.neuroimage.2012.02.043

29.

Rigoulot

Pell

M. D.

(2012). Seeing emotion with your ears: Emotional prosody implicitly guides visual attention to faces PLoS ONE, 7(1), e30740 https://doi.org/10.1371/journal.pone.0030740

30.

Sauter

D. A.

(2017). The nonverbal communication of positive emotions: An emotion family approach. Emotion Review, 9, 222-234. https://doi.org/10.1177/1754073916667236

31.

Sauter

D. A.

Scott

S. K.

(2007). More than one kind of happiness: Can we recognize vocal expressions of different positive states? Motivation and Emotion, 31, 192–199. https://doi.org/10.1007/s11031-007-9065-x

32.

Scherer

K. R.

Banse

Wallbott

H. G.

(2001). Emotion inferences from vocal expression correlate across languages and cultures. Journal of Cross-cultural Psychology, 32, 76–92. https://doi.org/10.1177/0022022101032001009

33.

Scherer

K.R.

Banse

Wallbott

H.G.

Goldbeck

(1991). Vocal cues in emotion encoding and decoding. Motivation and Emotion, 15, 123–148. https://doi.org/10.1007/BF00995674

34.

Schirmer

Kotz

S. A.

(2003). ERP evidence for a sex-specific Stroop effect in emotional speech. Journal of cognitive neuroscience, 15(8), 1135–1148. https://doi.org/10.1162/089892903322598102

35.

Vroomen

Driver

Gelder

B. D.

(2001). Is cross-modal integration of emotional expressions independent of attentional resources? Cognitive, Affective, & Behavioral Neuroscience, 1(4), 382–387. https://doi.org/10.3758/CABN.1.4.382

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB