This study explored whether visual (body movements) or auditory (musical material) information dominates perceived and felt emotions when observing a music performance. In a musical analogue of the McGurk effect and extending work juxtaposing facial expressions and auditory information for sung intervals to an actual musical performance, participants either watched video clips in which musical material and performer’s body movements were congruent or they saw incongruent clips that combined musical material with body movements from another piece that differed in emotional valence. Subsequent ratings of perceived and felt emotions showed a significant interaction between music and body movements, indicating that both auditory and visual channels determine the emotional content of the performance for listeners.
BratticoE.AlluriV.BogertB.JacobsenT.VartiainenN.NieminenS.TervaniemiM. (2011). A functional fMRI study of happy and sad emotions in music with and without lyrics. Frontiers in Psychology, 2, 308.
2.
DahlS.FribergA. (2004). Expressiveness of a marimba player’s body movements. TMH-QPSR, 46, 75–86.
3.
DahlS.FribergA. (2007). Visual perception of expressiveness in musicians’ body movements. Music Perception, 24, 433–454.
4.
Dalla BellaS.PeretzI.RousseauL.GosselinN. (2001). A developmental study of the affective value of tempo and mode in music. Cognition, 80, 1–10.
5.
DavidsonJ. W. (1993). Visual perception of performance manner in the movements of solo musicians. Psychology of Music, 21, 103–113.
6.
DePauloB. M.RosenthalR.EisenstatR. A.RogersP. L.FinkelsteinS. (1978). Decoding discrepant nonverbal cues. Journal of Personality and Social Psychology, 36, 313–323.
7.
DriverJ.SpenceC. (2000). Multisensory perception: Beyond modularity and convergence. Current Biology, 10, R731–R735.
8.
EkmanP.FriesenW. V.O’SullivanM.ChanA.Diacoyanni-TarlatzisI.HeiderK.. . .TzavarasA. (1987). Universals and cultural differences in the judgments of facial expressions of emotion. Journal of Personality and Social Psychology, 53, 712–717.
9.
FritzT.JentschkeS.GosselinN.SammlerD.PeretzI.TurnerR.. . .KoelschS. (2009). Universal recognition of three basic emotions in music. Current Biology, 19, 1–4.
10.
GagnonL.PeretzI. (2003). Mode and tempo relative contributions to “happy-sad” judgements in equitone melodies. Cognition and Emotion, 17, 25–40.
11.
JesseA.MassaroD. W. (2010). Seeing a singer helps comprehension of the song’s lyrics. Psychonomic Bulletin & Review, 17, 323–328.
12.
KhalfaS.PeretzI.BlondinJ.-P.RobertM. (2002). Event-related skin conductance responses to musical emotions in humans. Neuroscience Letters, 328, 145–149.
13.
LundqvistL.-O.CarlssonF.HilmerssonP.JuslinP. N. (2009). Emotional responses to music: Experience, expression, and physiology. Psychology of Music, 37, 61–90.
14.
McGurkH.MacDonaldJ. (1976). Hearing lips and seeing voices. Nature, 264, 746–748.
15.
SchererK. R.ZentnerM. R. (2001). Emotional effects of music: Production rules. In JuslinP. N.SlobodaJ. A. (Eds.), Music and emotion: Theory and research (pp. 361–392). Oxford, UK: Oxford University Press.
16.
SchutzM. (2008). Seeing music? What musicians need to know about vision. Empirical Musicology Review, 3, 83–108.
ThompsonW. F.GrahamP.RussoF. A. (2005). Seeing music performance: Visual influences on perception and experience. Semiotica, 156, 177–201.
19.
ThompsonW. F.RussoF. A.LivingstoneS. R. (2010). Facial expressions of singers influence perceived pitch relations. Psychonomic Bulletin & Review, 17, 317–322.
20.
ThompsonW. F.RussoF. A.QuintoL. (2008). Audio-visual integration of emotional cues in song. Cognition & Emotion, 22, 1–14.
21.
Van ZijlA. G. W.LuckG. (2013). Moved through music: The effect of experienced emotions on performers’ movement characteristics. Psychology of Music, 41, 175–197.
22.
VinesB. W.KrumhanslC. L.WanderleyM. M.DalcaI. M.LevitinD. J. (2005). Dimensions of emotion in expressive musical performance. Annals of the New York Academy of Sciences, 1060, 462–466.
23.
VinesB. W.KrumhanslC. L.WanderleyM. M.LevitinD. J. (2006). Cross-modal interactions in the perception of musical performance. Cognition, 101, 80–113.
24.
WanderleyM. M. (2002). Quantitative analysis of non-obvious performer gestures. In WachsmuthI.SowaT. (Eds.), Gesture and sign-language in human-computer interaction (pp. 241–253). Berlin, Germany: Springer Verlag.
25.
WanderleyM. M.VinesB. W.MiddletonN.McKayC.HatchW. (2005). The musical significance of clarinetists’ ancillary gestures: An exploration of the field. Journal of New Music Research, 34, 97–113.
26.
ZentnerM.GrandjeanD.SchererK. R. (2008). Emotions evoked by the sound of music: Characterization, classification, and measurement. Emotion, 8, 495–521.
