This study investigates cross-cultural vocal emotion recognition in a corpus with an affectively and linguistically balanced design. It has two main goals, one theoretical and the other methodological. First, it aims to explore the recognition of emotions in two typologically different languages, Dutch and Korean, within and across cultures. Second, it aims to contribute to the methodological development of the study of cross-cultural vocal emotion recognition by presenting a new corpus for Dutch and Korean emotional speech (the Demo/Koremo corpus), containing portrayals of eight emotions differing in arousal, valence, and basicness (joy, pride, tenderness, relief, anger, fear, sadness, irritation) produced by Dutch and Korean actors, and communicated in a single pseudo phrase which was viable in both languages. Dutch and Korean participants listened to recordings of all emotions produced by the Dutch and Korean actors and indicated for each one which emotion they thought it expressed. Both groups of listeners recognized emotions significantly above chance in both languages, but more accurately in their native language, in line with the Dialect Theory of emotion. Low-arousal emotions, negative emotions, and basic emotions were recognized more accurately than their counterparts. While some of these results replicate earlier findings, others—the effect of arousal and the within-cultural effect of valence and basicness—had not been previously investigated. This study provides new insights in cross-cultural vocal emotion recognition and contributes to the methodological toolkit of intercultural emotion recognition research.
AlbasD. C.McCluskeyK. W.AlbasC. A. (1976). Perception of the emotional content of speech: A comparison of two Canadian groups. Journal of Cross-Cultural Psychology, 7(4), 481–490. https://doi.org/10.1177/002202217674009
2.
BaileyW.NowickiS.ColeS. P. (1998). The ability to decode nonverbal information in African American, African and Afro-Caribbean, and European American adults. Journal of Black Psychology, 24(4), 418–431. https://doi.org/10.1177/00957984980244002
3.
BanseR.SchererK. R. (1996). Acoustic profiles in vocal emotion expression. Journal of Personality and Social Psychology, 70(3), 614–636. https://doi.org/10.1037/0022-3514.70.3.614
4.
BänzigerT.MortillaroM.SchererK. R. (2012). Introducing the Geneva Multimodal expression corpus for experimental research on emotion perception. Emotion, 12(5), 1161.
5.
BänzigerT.SchererK. R. (2007). Using actor portrayals to systematically study multimodal emotion expression: The GEMEP corpus. In PaivaA. C. R.PradaR.PicardR. W. (Eds.), International Conference on Affective Computing and Intelligent Interaction (pp. 476–487). Springer.
6.
BarrD.LevyR.ScheepersC.TilyH. (2013). Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68(3), 255–278. https://doi.org/10.1016/j.jml.2012.11.001
7.
BatesD.MächlerM.BolkerB. M.WalkerS. C. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48. https://doi.org/10.18637/jss.v067.i01
8.
BeierE.ZautraA. J. (1972). The identification of vocal expressions of emotion across cultures. Journal of Consulting and Clinical Psychology, 40(4), 560.
9.
BoersmaP. (2001). Praat, a system for doing phonetics by computer. Glot International, 5, 341–345.
BreitensteinC.van LanckerD.DaumI. (2001). The contribution of speech rate and pitch variation to the perception of vocal emotions in a German and an American sample. Cognition and Emotion, 15(1), 57–79. https://doi.org/10.1080/0269993004200114
12.
BryantG. A.BarrettH. C. (2008). Vocal emotion recognition across disparate cultures. Journal of Cognition and Culture, 8(1–2), 135–148. https://doi.org/10.1163/156770908X289242
13.
ChronakiG.WigelsworthM.PellM. D.KotzS. A. (2018). The development of cross-cultural recognition of vocal emotion during childhood and adolescence. Scientific Reports, 8(1), 1–17. https://doi.org/10.1038/s41598-018-26889-1
14.
ChungS. J. (1999). Vocal expression and perception of emotion in Korean. In OhalaJ. J.HasegawaY.OhalaM.GranvilleD.BaileyA. C. (Eds.), Proceedings of the 14th International Conference of Phonetic Sciences (ICPhS), San Francisco, CA, United States (pp. 969–972).
15.
ClarkM. S.MilbergS.ErberR. (1984). Effects of arousal on judgments of others’ emotions. Journal of Personality and Social Psychology, 46(3), 551–560. https://doi.org/10.1037/0022-3514.46.3.551
16.
CordaroD. T.KeltnerD.TsheringS.WangchukD.FlynnL. M. (2016). The voice conveys emotion in ten globalized cultures and one remote village in Bhutan. Emotion, 16(1), 117–128. https://doi.org/10.1037/emo0000100
17.
CowenA. S.KeltnerD. (2017). Self-report captures 27 distinct categories of emotion bridged by continuous gradients. Proceedings of the National Academy of Sciences, 114(38), E7900–E7909. https://doi.org/10.1073/pnas.1702247114
18.
CowenA. S.LaukkaP.ElfenbeinH. A.LiuR.KeltnerD. (2019). The primacy of categories in the recognition of 12 emotions in speech prosody across two cultures. Nature Human Behaviour, 3(4), 369–382. https://doi.org/10.1038/s41562-019-0533-6
19.
CutlerA.Van DonselaarW. (2001). Voornaam is not (really) a homophone: Lexical prosody and lexical access in Dutch. Language and Speech, 44(2), 171–195. https://doi.org/10.1177/00238309010440020301
20.
DarwinC. (1998). The expression of the emotions in man and animals (3rd ed.). John Murray. (Original work published 1872)
21.
EkmanP. (1972). Universals and cultural differences in facial expressions of emotion. In ColeJ. K. (Ed.), Nebraska Symposium on Motivation (Vol. 19, pp. 207–283). University of Nebraska Press.
ElfenbeinH. A.AmbadyN. (2002). On the universality and cultural specificity of emotion recognition: A meta-analysis. Psychological Bulletin, 128(2), 203–235. https://doi.org/10.1037/0033-2909.128.2.203
30.
ElfenbeinH. A.BeaupréM.LévesqueM.HessU. (2007). Toward a dialect theory: Cultural differences in the expression and recognition of posed facial expressions. Emotion, 7(1), 131–146. https://doi.org/10.1037/1528-3542.7.1.131
31.
Feldman BarrettL. (2017). How emotions are made: The secret life of the brain. Pan Macmillan.
32.
Feldman BarrettL.RussellJ. A. (2014). The psychological construction of emotion. Guilford Publications.
33.
GendronM.CrivelliC.Feldman BarrettL. (2018). Universality reconsidered: Diversity in making meaning of facial expressions. Current Directions in Psychological Science, 27(4), 211–219. https://doi.org/10.1177/0963721417746794
34.
GoudbeekM.BroersmaM. (2010a). The Demo/Kemo corpus: A principled approach to the study of cross-cultural differences in the vocal expression and perception of emotion. In CalzolariN.ChoukriK.MaegaardB.MarianiJ.OdijkJ.PiperidisS.RosnerM.TapiasD. (Eds.), Proceedings of the 7th International Conference on Language Resources and Evaluation (LREC 2010), Valletta, Malta (pp. 2211–2215).
35.
GoudbeekM.BroersmaM. (2010b). Language specific effects of emotion on phoneme duration. In KobayashiT.HiroseK.NakamuraS. (Eds.), Proceedings of the 11th Annual Conference of the International Speech Communication Association (Interspeech 2010), Chiba, Japan (pp. 2026–2029).
36.
GoudbeekM.SchererK. R. (2010). Beyond arousal: Valence and potency/control cues in the vocal expression of emotion. The Journal of the Acoustical Society of America, 128(3), 1322. https://doi.org/10.1121/1.3466853
37.
GussenhovenC. (1993). The Dutch foot and the chanted call. Journal of Linguistics, 29(1), 37–63. https://doi.org/https://www.jstor.org/stable/pdf/4176207
38.
GussenhovenC. (2005). Transcription of Dutch intonation. In In Sun-Ah JunP. (Ed.), The phonology of intonation and phrasing (pp. 118–145). Oxford University Press. https://doi.org/10.1093/acprof
39.
HarreR. (1986). The social construction of emotions. Blackwell.
40.
HuangC. F.EricksonD.AkagiM. (2008). Comparison of Japanese expressive speech perception by Japanese and Taiwanese listeners. In Proceedings European Conference on Noise Control (pp. 2317–2322). https://doi.org/10.1121/1.2933803
41.
JensenT. W. (2014). Emotion in languaging: Languaging as affective, adaptive, and flexible behavior in social interaction. Frontiers in Psychology, 5, 1–14. https://doi.org/10.3389/fpsyg.2014.00720
42.
JiangX.PaulmannS.RobinJ.PellM. D. (2015). More than accuracy: Nonverbal dialects modulate the time course of vocal emotion recognition across cultures. Journal of Experimental Psychology: Human Perception and Performance, 41(3), 597–612. https://doi.org/10.1037/xhp0000043
JunS. A. (2006). Intonational phonology of Seoul Korean revisited. Japanese/Korean Linguistics, 14, 15–26.
45.
JürgensR.DroletM.PirowR.ScheinerE.FischerJ. (2013). Encoding conditions affect recognition of vocally expressed emotions across cultures. Frontiers in Psychology, 4, 1–10. https://doi.org/10.3389/fpsyg.2013.00111
46.
JuslinP. N.LaukkaP. (2003). Communication of emotions in vocal expression and music performance: Different channels, same code?Psychological Bulletin, 129(5), 770–814. https://doi.org/10.1037/0033-2909.129.5.770
KeltnerD.SauterD.TracyJ.CowenA. (2019). Emotional expression: Advances in basic emotion theory. Journal of Nonverbal Behavior, 43, 133–160.
49.
KimJ.DavisC.CutlerA. (2008). Perceptual tests of rhythmic similarity: II. Syllable rhythm. Language and Speech, 51(4), 343–359.
50.
KlattD. H.KlattL. C. (1990). Analysis, synthesis, and perception of voice quality variations among female and male talkers. The Journal of the Acoustical Society of America, 87(2), 820–857. https://doi.org/10.1121/1.398894
51.
KramerE. (1964). Elimination of verbal cues in judgments of emotion from voice. Journal of Abnormal and Social Psychology, 68(4), 390–396. https://doi.org/10.1037/h0042473
52.
LaneR. D.ChuaP. M. L.DolanR. J. (1999). Common effects of emotional valence, arousal and attention on neural activation during visual processing of pictures. Neuropsychologia, 37(9), 989–997. https://doi.org/10.1016/S0028-3932(99)00017-2
53.
LaukkaP. (2003). Categorical perception of emotion in vocal expression. Annals of the New York Academy of Sciences, 1000, 283–287. https://doi.org/10.1196/annals.1280.026
54.
LaukkaP.ElfenbeinH. A. (2021). Cross-cultural emotion recognition and in-group advantage in vocal expression: A meta-analysis. Emotion Review, 13(1), 3–11. https://doi.org/10.1017/S0272263120000674
55.
LaukkaP.ElfenbeinH. A.SöderN.NordströmH.AlthoffJ.ChuiW.IrakiF. K.RockstuhlT.ThingujamN. S. (2013). Cross-cultural decoding of positive and negative non-linguistic emotion vocalizations. Frontiers in Psychology, 4, 1–8. https://doi.org/10.3389/fpsyg.2013.00353
56.
LaukkaP.JuslinP. N.BresinR. (2005). A dimensional approach to vocal expression of emotion. Cognition and Emotion, 19(5), 633–653. https://doi.org/10.1080/02699930441000445
57.
LaukkaP.ThingujamN. S.IrakiF. K.ElfenbeinH. A.RockstuhlT.ChuiW.AlthoffJ. (2016). The expression and recognition of emotions in the voice across five nations: A lens model analysis based on acoustic features. Journal of Personality and Social Psychology, 111(5), 686–705. https://doi.org/10.1037/pspi0000066
MandalM. K. (2008). Cultural in-group advantage in accuracy at recognizing vocal expressions of emotion. Psychological Studies, 53, 126–132.
60.
MarinoL. (2017). Thinking chickens: A review of cognition, emotion, and behavior in the domestic chicken. Animal Cognition, 20(2), 127–147.
61.
MatsumotoD. (2002). Methodological requirements to test a possible in-group advantage in judging emotions across cultures: Comment on Elfenbein and Ambady (2002) and evidence. Psychological Bulletin, 128(2), 236–242. https://doi.org/10.1037/0033-2909.128.2.236
MesquitaB.FrijdaN. H. (1992). Cultural variations in emotions: A review. Psychological Bulletin, 112(2), 179–204.
64.
Mourão-MirandaJ.VolchanE.MollJ.De Oliveira-SouzaR.OliveiraL.BramatiI.GattassR.PessoaL. (2003). Contributions of stimulus valence and arousal to visual activation during emotional perception. NeuroImage, 20(4), 1955–1963. https://doi.org/10.1016/j.neuroimage.2003.08.011
65.
PaulmannS.UskulA. K. (2014). Cross-cultural emotional prosody recognition: Evidence from Chinese and British listeners. Cognition and Emotion, 28(2), 230–244. https://doi.org/10.1080/02699931.2013.812033
66.
PellM. D.MonettaL.PaulmannS.KotzS. A. (2009). Recognizing Emotions in a Foreign Language. Journal of Nonverbal Behavior, 33(2), 107–120. https://doi.org/10.1007/s10919-008-0065-7
67.
PessoaL. (2015). Précis on the cognitive-emotional brain. Behavioral and Brain Sciences, 38, e71.
68.
RaheelA.AnwarS. M.MajidM. (2019). Emotion recognition in response to traditional and tactile enhanced multimedia using electroencephalography. Multimedia Tools and Applications, 78(10), 13971–13985.
69.
R Core Team. (2018). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.r-project.org
70.
RussellJ. A. (1994). Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies. Psychological Bulletin, 115(1), 102–141.
SacharinV.SchlegelK.SchererK. R. (2012). Geneva emotion wheel rating study. Center for Person, Kommunikation, Aalborg University, NCCR Affective Sciences.
73.
SauterD. A.EisnerF.CalderA. J.ScottS. K. (2010). Perceptual cues in nonverbal vocal expressions of emotion. Quarterly Journal of Experimental Psychology, 63(11), 2251–2272. https://doi.org/10.1080/17470211003721642
74.
SauterD. A.ScottS. K. (2007). More than one kind of happiness: Can we recognize vocal expressions of different positive states?Motivation and Emotion, 31(3), 192–199. https://doi.org/10.1007/s11031-007-9065-x
75.
SchererK. R. (2003). Vocal communication of emotion: A review of research paradigms. Speech Communication, 40(1–2), 227–256.
SchererK. R. (2009). The dynamic architecture of emotion: Evidence for the component process model. Cognition & Emotion, 23(7), 1307–1351. https://doi.org/10.1080/02699930902928969
78.
SchererK. R. (2019). Acoustic patterning of emotion vocalizations. In FrühholzS.BelinP. (Eds.), Oxford handbook of voice perception (pp. 61–91). Oxford University Press.
79.
SchererK. R.BanseR.WallbottH. G. (2001). Emotion inferences from vocal expression correlate across languages and cultures. Journal of Cross-Cultural Psychology, 32(1), 76–92. https://doi.org/10.1177/0022022101032001009
80.
SchererK. R.Clark-PolnerE.MortillaroM. (2011). In the eye of the beholder? Universality and cultural specificity in the expression and perception of emotion. International Journal of Psychology, 46(6), 401–435. https://doi.org/10.1080/00207594.2011.626049
81.
SchererS.SiegertI.BigalkeL.MeudtS. (2010). Developing an expressive speech labeling tool incorporating the temporal characteristics of emotion. In CalzolariN.ChoukriK.MaegaardB.MarianiJ.OdijkJ.PiperidisS.RosnerM.TapiasD. (Eds.), Proceedings of the 7th International Conference on Language Resources and Evaluation, LREC 2010, Valletta, Malta (pp. 1172–1175).
82.
ShiotaM. N.CamposB.KeltnerD.HertensteinM. J. (2004). Positive emotion and the regulation of interpersonal relationships. In PhilippotP.FeldmanR. S. (Eds.), The regulation of emotion (pp. 127–155). Lawrence Erlbaum.
83.
ShiotaM. N.CamposB.OveisC.HertensteinM. J.Simon-ThomasE.KeltnerD. (2017). Beyond happiness: Building a science of discrete positive emotions. American Psychologist, 72(7), 617. https://doi.org/https://doi.org/10.1037/a0040456
84.
ShochiT.RilliardA.AubergéV. (2009). Intercultural perception of English, French. The Role of Prosody in Affective Speech, 97, 31.
85.
SmithS. D.McIverT. A.Di NellaM. S. J.CreaseM. L. (2011). The effects of valence and arousal on the emotional modulation of time perception: Evidence for multiple stages of processing. Emotion, 11(6), 1305–1313. https://doi.org/10.1037/a0026145
86.
StanislavskiK. (1988). An actor prepares. Methuen. (Original work published 1936)
TracyJ. L.RobinsR. W. (2007). Emerging insights into the nature and function of pride. Current Directions in Psychological Science, 16(3), 147–150. https://doi.org/10.1111/j.1467-8721.2007.00493.x
WagnerH. L. (1993). On measuring performance in category judgment studies of nonverbal behavior. Journal of Nonverbal Behavior, 17(1), 3–28. https://doi.org/10.1007/BF00987006
