Previous studies on multisensory integration (MSI) of musical emotions have yielded inconsistent results. The distinct features of the music materials and different musical expertise levels of participants may account for that. This study aims to explore the neural mechanism for the audio-visual integration of musical emotions and infer the reasons for inconsistent results in previous studies by investigating the influence of the type of musical emotions and musical training experience on the mechanism. This fMRI study used a block-design experiment. Music excerpts were selected to express fear, happiness, and sadness, presented under audio only (AO) and audio-visual (AV) modality conditions. Participants were divided into two groups: one comprising musicians who had been musically trained for many years and the other non-musicians with no musical expertise. They assessed the type and intensity of musical emotion after listening to or watching excerpts. Brain regions related to MSI of emotional information and default mode network (DMN) are sensitive to sensory modality conditions and emotion-type changes. Participants in the non-musician group had more, and bilateral distribution of brain regions showed greater activation in the AV assessment stage. By contrast, the musician group had less and lateralized right-hemispheric distribution of brain regions.
AmmiranteP.ThompsonW. F. (2012). Continuation tapping to triggered melodies: Motor resonance effects of melodic motion. Experimental Brain Research, 216(1), 51–60.
2.
Andrews-HannaJ. R. (2012). The brain’s default network and its adaptive role in internal mentation. Neuroscientist, 18(3), 251–270.
3.
AubéW.Angulo-PerkinsA.PeretzI.ConchaL.ArmonyJ. L. (2015). Fear across the senses: Brain responses to music, vocalizations and facial expressions. Social Cognitive & Affective Neuroscience, 10(3), 399–407.
4.
BogertB.Numminen-KonttiT.GoldB.SamsM.NumminenJ.BurunatI.LampinenJ.BratticoE. (2016). Hidden sources of joy, fear, and sadness: Explicit versus implicit neural processing of musical emotions. Neuropsychologia, 89, 393–402.
5.
BucknerR. L.Andrews-HannaJ. R.SchacterD. L. (2008). The brain’s default network: Anatomy, function, and relevance to disease. Annals of the New York Academy of Sciences, 1124, 1–38.
6.
CaiY. J.PanX. F.ZhuangZ. C. X. (2007). An experimental research on how 8 music excerpts’ tempo and melody influenced undergraduates’ emotion. Psychological Science, 30(1), 196–198.
7.
Chaddock-HeymanL.LouiP.WengT. B.WeisshappelR.KramerA. F. (2021). Musical training and brain volume in older adults. Brain Sciences, 11(1), 50.
8.
ChapadosC.LevitinD. J. (2008). Cross-modal interactions in the experience of musical performances: Physiological correlates. Cognition, 108(3), 639–651.
9.
ChristensenJ. F.GaiggS. B.AntoniG.PeterO.BeatrizC. M. (2014). Enhancing emotional experiences to dance through music: The role of valence and arousal in the cross-modal bias. Frontiers in Human Neuroscience, 8, Article 757.
10.
CorinnaP.LorenzD.Jan-HendrikB.SchlochtermeierL. H.HermannK.JacobsA. M.FritzT. H.KoelschS.KuchinkeL. (2014). How music alters a kiss: Superior temporal gyrus controls fusiform-amygdalar effective connectivity. Social Cognitive & Affective Neuroscience, 11, 1770–1778.
11.
DahlS.FribergA. (2007). Visual perception of expressiveness in musicians’ body movements. Music Perception, 24(5), 433–454.
12.
DavidsonJ. W. (2012). Bodily movement and facial actions in expressive musical performance by solo and duo instrumentalists: Two distinctive case studies. Psychology of Music, 40(5), 595–633.
13.
DellacherieD.RoyM.HuguevilleL.PeretzI.SamsonS. (2011). The effect of musical experience on emotional self-reports and psychophysiological responses to dissonance. Psychophysiology, 48(3), 337–349.
GabrielssonA. (2002). Emotion perceived and emotion felt: Same or different?Musicae Scientiae, 5(1), 123–147.
16.
HolmesN. P. (2009). The principle of inverse effectiveness in multisensory integration: Some statistical considerations. Brain Topography, 21(3), 168–176.
17.
HunterP. G.SchellenbergE. G.SchimmackU. (2010). Feelings and perceptions of happiness and sadness induced by music: Similarities, differences, and mixed emotions. Psychology of Aesthetics, Creativity, and the Arts, 4(1), 47–56.
18.
IlieG.ThompsonW. F. (2006). A comparison of acoustic cues in music and speech for three dimensions of affect. Music Perception: An Interdisciplinary Journal, 23(4), 319–330.
19.
JeongJ. W.DiwadkarV. A.ChuganiC. D.SinsoongsudP.MuzikO.BehenM. E.ChuganiH. T.ChuganiD. C. (2011). Congruence of happy and sad emotion in music and faces modifies cortical audiovisual activation. NeuroImage, 54(4), 2973–2982.
20.
JomoriI.HoshiyamaM.UemuraJ.NakagawaY.HoshinoA.IwamotoY. (2013). Effects of emotional music on visual processes in inferior temporal area. Cognitive Neuroscience, 4(1), 21–30.
21.
JuslinP. N.VästfjällD. (2008). Emotional responses to music: The need to consider underlying mechanisms. Behavioral & Brain Sciences, 31(5), 575–621.
22.
Kantor-MartynuskaJ.BigandE. (2013). Individual differences in granularity of the affective responses to music. Polish Psychological Bulletin, 44(4), 399–408.
23.
Kantor-MartynuskaJ.HorabikJ. (2015). Granularity of emotional responses to music: The effect of musical expertise. Psychology of Aesthetics, Creativity, and the Arts, 9(3), 235–247.
24.
KarinP.FrancesC.CarolS.PollickF. E. (2011). The music of your emotions: Neural substrates involved in detection of emotional correspondence between auditory and visual music actions. PLOS ONE, 6(4), Article e19165.
25.
KawakamiA.FurukawaK.KatahiraK.OkanoyaK. (2013). Sad music induces pleasant emotion. Frontiers in Psychology, 4, Article 311.
26.
Korsakova-KreynM. (2018). Two-level model of embodied cognition in music. Psychomusicology: Music, Mind, and Brain, 28(4), 240–259.
27.
KrahéC.HahnU.WhitneyK. (2015). Is seeing (musical) believing? the eye versus the ear in emotional responses to music. Psychology of Music, 43(1), 140–148.
28.
KretM. E.RoelofsK.StekelenburgJ. J.GelderB. D. (2013). Emotional signals from faces, bodies and scenes influence observers’ face expressions, fixations and pupil-size. Frontiers in Human Neuroscience, 7, Article 810.
29.
KrumhanslC. L. (2002). Music: A link between cognition and emotion. Current Directions in Psychological Science, 11(2), 45–50.
30.
LawL. N. C.ZentnerM. (2012). Assessing musical abilities objectively: Construction and validation of the profile of music perception skills. PLOS ONE, 7(12), Article e52508.
31.
LemanM.MaesP. J. (2015). The role of embodiment in the perception of music. Empirical Musicology Review, 9(3–4), 236–246.
32.
LiP.ZhangM. M.LiS. X.ZhangH. Y.LuoW. B. (2019). The integration of facial expression and vocal emotion and its brain mechanism. Advances in Psychological Science, 27(7), 1205–1214.
33.
LiY.ShuH. (2014). The brain mechanisms and functional hypothesis of default mode network and its clinical application. Advances in Psychological Science, 22(2), 234–249.
34.
LivingstoneR. S.ThompsonW. F. (2009). The emergence of music from the Theory of Mind. Musicae Scientiae, 13(2), 83–115.
35.
MaesP. J.LemanM.PalmerC.WanderleyM. M. (2014). Action based effects on music perception. Frontiers in Psychology, 4, Article 1008.
36.
MitchellH. F.MacDonaldR. A. R. (2014). Listeners as spectators? Audio-visual integration improves music performer identification. Psychology of Music, 42(1), 112–127.
37.
Molnar-SzakacsI.OveryK. (2006). Music and mirror neurons: From motion to emotion. Social Cognitive & Affective Neuroscience, 1(3), 235–241.
38.
MostT.AmirN.DotanG.WeiselA. (2008). Auditory and visual aspects of emotion production by children and adults. Journal of Speech & Language Pathology Applied Behavior Analysis, 3(1), 86–96.
39.
NordströmH.LaukkaP. (2019). The time course of emotion recognition in speech and music. The Journal of the Acoustical Society of America, 145(5), 3058–3074.
40.
PandeyaY. R.LeeJ. (2021). Deep learning-based late fusion of multimodal information for emotion classification of music video. Multimedia Tools and Applications, 80(38), 1–19.
41.
PaquetteS.TakerkartS.SagetS.PeretzI.BelinP. (2018). Cross-classification of musical and vocal emotions in the auditory cortex: Cross-classification of musical and vocal emotions. Annals of the New York Academy of Sciences, 1423, 329–337.
42.
ParkJ. Y.GuB. M.KangD. H.ShinY. W.ChoiC. H.LeeJ. M.KwonJ. S. (2010). Integration of cross-modal emotional information in the human brain: An fMRI study. Cortex, 46(2), 161–169.
43.
PetriniK.McaleerP.PollickF. (2010). Audiovisual integration of emotional signals from music improvisation does not depend on temporal correspondence. Brain Research, 1323(8), 139–148.
44.
PlatzF.KopiezR. (2012). When the eye listens: A meta-analysis of how audio-visual presentation enhances the appreciation of music performance. Music Perception, 30, 71–83.
45.
ProverbioA. M.De BenedettoF. (2018). Auditory enhancement of visual memory encoding is driven by emotional content of the auditory material and mediated by superior frontal cortex. Biological Psychology, 29, 164–175.
46.
Román-CaballeroR.ArnedoM.TrivioM.LupiáezJ. (2018). Musical practice as an enhancer of cognitive function in healthy aging—A systematic review and meta-analysis. PLOS ONE, 13(11), Article e0207957.
47.
SachsM. E.HabibiA.DamasioA.KaplanJ. T. (2018). Decoding the neural signatures of emotions expressed through sound. NeuroImage, 174, 1–10.
48.
SandstromG. M.RussoF. A. (2013). Absorption in music: Development of a scale to identify individuals with strong emotional responses to music. Psychology of Music, 41(2), 216–228.
49.
SchubertE. (2007). Locus of emotion: The effect of task order and age on emotion perceived and emotion felt in response to music. Journal of Music Therapy, 44(4), 344–368.
50.
ShodaH.AdachiM. (2016). Expressivity, affective nuance, and presentation modality in a performer-to-audience communication. Psychomusicology: Music, Mind, and Brain, 26(2), 167–178.
51.
SigurdssonT.DoyereV.CainC. K.LeDouxJ. E. (2007). Long-term potentiation in the amygdala: A cellular mechanism of fear learning and memory. Neuropharmacology, 52, 215–227.
52.
SpitzerM.CoutinhoE. (2014). The effects of expert musical training on the perception of emotions in Bach’s Sonata for Unaccompanied Violin No. 1 in G Minor (BWV 1001). Psychomusicology: Music, 24(1), 35–57.
53.
SteinB. E.StanfordT. R.RamachandranR.PerraultT. J.RowlandB. A. (2009). Challenges in quantifying multisensory integration: Alternative criteria, models, and inverse effectiveness. Experimental Brain Research, 198(2–3), 113–126.
54.
SymonsA. E.El-DeredyW.SchwartzeM.KotzS. A. (2016). The functional role of neural oscillations in non-verbal emotional communication. Frontiers in Human Neuroscience, 10, Article 239.
55.
TaruffiL.KoelschS. (2014). The paradox of music-evoked sadness: An online survey. PLOS ONE, 9(10), Article e110490.
56.
ThompsonW. F.RussoF. A.QuintoL. (2008). Audio-visual integration of emotional cues in song. Cognition and Emotion, 22, 1457–1470.
57.
VesaP.SanazN. F.KerttuS.TomiK.LihuaS.KarlssonH. K., et al. (2020). Decoding music-evoked emotions in the auditory and motor cortex. Cerebral Cortex. 31(5), 2549–2560.
58.
VinesB. W.KrumhanslC. L.WanderleyM. M.DalcaI. M.LevitinD. J. (2011). Music to my eyes: Cross-modal interactions in the perception of emotions in musical performance. Cognition, 118(2), 157–170.
59.
VuoskoskiJ. K.EerolaT. (2015). Extramusical information contributes to emotions induced by music. Psychology of Music, 43(2), Article 80850V.
60.
VuoskoskiJ. K.GattiE.SpenceC.ClarkeE. F. (2016). Do visual cues intensify the emotional responses evoked by musical performance? a psychophysiological investigation. Psychomusicology: Music, Mind, and Brain, 26(2), 179–188.
61.
VuoskoskiJ. K.ThompsonM. R.ClarkeE. F.SpenceC. (2014). Crossmodal interactions in the perception of expressivity in musical performance. Attention, Perception, & Psychophysics, 76(2), 591–604.
62.
VuoskoskiJ. K.ThompsonW. F.McilwainD.EerolaT. (2012). Who enjoys listening to sad music and why?Music Perception: An Interdisciplinary Journal, 29(3), 311–317.
63.
WeijkampJ.SadakataM. (2017). Attention to affective audio-visual information: Comparison between musicians and non-musicians. Psychology of Music, 45, 204–215.
64.
WenX. H.LiuQ.SunH. J.ZhangQ. L.HaoM. J.MouH. R. (2009). Science theoretical models of multisensory cues integration. Advances in Psychological, 17(4), 659–666.
65.
WilliamsL. M.MathersulD.PalmerD. M.GurR. C.GurR. E.GordonE. (2009). Explicit identification and implicit recognition of facial emotions: I. age effects in males and females across 10 decades. Journal of Clinical & Experimental Neuropsychology, 31(3), 257–277.
66.
WöllnerC. (2012). Is empathy related to the perception of emotional expression in music? A multimodal time-series analysis. Psychology of Aesthetics, Creativity, and the Arts, 6(3), 214–223.
67.
YangH.HouC. Y.LuJ.LuoC.YaoD. Z. (2016). Functional brain network study on resting state of composers. Chinese Journal of Biomedical Engineering, 35(5), 612–615.
