Investigating emotional responses to self-selected sad music via self-report and automated facial analysis

Abstract

People often listen to sad music in spite of its seemingly negative qualities. Sad music, and especially sad music with a personal significance, has been shown to evoke a wide span of emotions with both positive and negative qualities. We compared emotional responses to familiar self-selected sad music (SSSM) with both unfamiliar sad and unfamiliar happy music. Alongside self-reports, a commercial, continuous measure of discrete facial expressions was applied, promising an in-depth assessment of both the quality and strength of experienced affective states at any given point in time. Results of the facial analysis showed that SSSM evoked more mixed affective states than unfamiliar sad music. Also, listeners reacted with consistent facial expressions to distinct musical events, e.g. the introduction of a lead voice. SSSM evoked more self-reported feelings of nostalgia, reminiscence, being moved, and chills and tears than unfamiliar sad and happy music. Furthermore, SSSM resulted in more self-reported happiness and a similar trend with happy facial expressions compared to unfamiliar sad music. These results point to the emotional diversity and the strong involvement of positive affective states elicited by SSSM, even when compared with music of similar quality, such as unfamiliar sad music. Automated facial analysis allows us to observe emotions on a more detailed level in terms of time resolution, onset, intensity and concurrence of discrete affective states. This technique is promising for future research, particularly when investigating mixed emotions and the social aspect of emotions in response to music.

Keywords

affective responses basic emotion continuous evaluation procedure (CEP)emotions FaceReader FACS facial expressions mixed emotions nostalgia sad music self-report

Get full access to this article

View all access options for this article.

References

Ali

S. O.

Peynircioğlu

Z. F.

(2006). Songs and emotions: Are lyrics and melodies equal partners? Psychology of Music, 34(4), 511–534.

Barrett

F. S.

Grimm

K. J.

Robins

R. W.

Wildschut

Sedikides

Janata

(2010). Music-evoked nostalgia: Affect, memory, and personality. Emotion, 10(3), 390–403.

Beaupré

M. G.

Hess

(2005). Cross-cultural emotion recognition among Canadian ethnic groups. Journal of Cross-Cultural Psychology, 36(3), 355–370.

Brattico

Alluri

Bogert

Jacobsen

Vartiainen

Nieminen

S. K.

Tervaniemi

(2011). A functional MRI study of happy and sad emotions in music with and without lyrics. Frontiers in Psychology, 2, 308.

Brown

D. E.

(1991). Human universals. New York, NY: McGraw-Hill.

Cacioppo

J. T.

Petty

R. E.

Losch

M. E.

Kim

H. S.

(1986). Electromyographic activity over facial muscle regions can differentiate the valence and intensity of affective reactions. Journal of Personality and Social Psychology, 50(2), 260–268.

Carlson

J. G.

Hatfield

(1992). Psychology of emotion. San Diego, CA: Harcourt Brace Jovanovich.

Chentsova-Dutton

Y. E.

Tsai

J. L.

(2010). Self-focused attention and emotional reactivity: The role of culture. Journal of Personality and Social Psychology, 98(3), 507–519.

Clore

G. L.

Huntsinger

J. R.

(2007). How emotions inform judgment and regulate thought. Trends in Cognitive Sciences, 11(9), 393–399.

10.

Cohen

A. S.

Morrison

S. C.

Callaway

D. A.

(2013). Computerized facial analysis for understanding constricted/blunted affect: Initial feasibility, reliability, and validity data. Schizophrenia Research, 148(1–3), 111–116.

11.

Cootes

T. F.

Taylor

C. J.

(2000). Statistical models of appearance for computer vision [Technical report]. Manchester, UK: University of Manchester.

12.

Danner

Sidorkina

Joechl

Duerrschmid

(2013). Make a face! Implicit and explicit measurement of facial expressions elicited by orange juices using face reading technology. Food Quality and Preference, 32, 167–172.

13.

D’Arcey

J. T.

(2013). Assessing the validity of FaceReader using facial EMG (Unpublished master’s thesis). California State University, Chico.

14.

den Uyl

M. J.

van Kuilenburg

(2005). The FaceReader: Online facial expression recognition. In Noldus

L. P. J. J.

Grieco

Loijens

L.W. S.

Zimmerman

P.H.

(Eds.), Proceedings of Measuring Behavior, 5th International Conference on Methods and Techniques in Behavioral Research (pp. 589–590). Wageningen, The Netherlands: Noldus Information Technology.

15.

Derntl

Seidel

E.-M.

Kainz

Carbon

C.-C.

(2009). Recognition of emotional expressions is affected by inversion and presentation time. Perception, 38(12), 1849–1862.

16.

Dimberg

Thunberg

Elmehed

(2000). Unconscious facial reactions to emotional facial expressions. Psychological Science, 11(1), 86–89.

17.

Ekman

(1992). An argument for basic emotions. Cognition & Emotion, 6(3), 169–200.

18.

Ekman

(1999). Basic emotions. In Dalgleish

Power

(Eds.), Handbook of cognition and emotion (pp. 45–60). New York, NY: Wiley.

19.

Ekman

Friesen

W. V.

(1971). Constants across cultures in the face and emotion. Journal of Personality and Social Psychology, 17(2), 124–129.

20.

Ekman

Friesen

W. V.

(1978). Facial action coding system. Palo Alto, CA: Consulting Psychologists Press.

21.

Elfenbein

H. A.

Ambady

(2002). Is there an in-group advantage in emotion recognition? Psychological Bulletin, 128(2), 243–249.

22.

Frith

(2009). Role of facial expressions in social interactions. Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1535), 3453–3458.

23.

Gabrielsson

(2002). Emotion perceived and emotion felt: Same or different? Musicae Scientiae, 5(1 Suppl., Fall 2001–2002), 123–147.

24.

Garrido

(2009). Rumination and sad music: A review of the literature and a future direction. In Stevens

Schubert

Kruithof

Buckley

Fazio

(Eds.), Proceedings of the 2nd International Conference on Music Communication Science (ICoMCS2). Sydney, Australia: HCSNet, University of Western Sydney.

25.

Goldberg

J. H.

(2014). Measuring software screen complexity: Relating eye tracking, emotional valence, and subjective ratings. International Journal of Human-Computer Interaction, 30(7), 518–532.

26.

Grewe

Katzur

Kopiez

Altenmüller

(2011). Chills in different sensory domains: Frisson elicited by acoustical, visual, tactile and gustatory stimuli. Psychology of Music, 39(2), 220–239.

27.

Grewe

Kopiez

Altenmüller

(2009). Chills as an indicator of individual emotional peaks. Annals of the New York Academy of Sciences, 1169(1), 351–354.

28.

Grewe

Nagel

Kopiez

Altenmüller

(2007). Emotions over time: Synchronicity and development of subjective, physiological, and facial affective reactions to music. Emotion, 7(4), 774–788.

29.

Hager

J. C.

Ekman

Friesen

W. V.

(2002). Facial action coding system (2nd ed.). Salt Lake City, UT: A Human Face.

30.

Hazlett

R. L.

Hazlett

S. Y.

(1999). Emotional response to television commercials: Facial EMG vs. self-report. Journal of Advertising Research, 39, 7–24.

31.

Hietanen

J. K.

Surakka

Linnankoski

(1998). Facial electromyographic responses to vocal affect expressions. Psychophysiology, 35(5), 530–536.

32.

Hunter

P. G.

Schellenberg

E. G.

Schimmack

(2010). Feelings and perceptions of happiness and sadness induced by music: Similarities, differences, and mixed emotions. Psychology of Aesthetics, Creativity, and the Arts, 4(1), 47–56.

33.

Huron

(2001). Is music an evolutionary adaptation? Annals of the New York Academy of Sciences, 930(1), 43–61.

34.

Huron

(2011). Why is sad music pleasurable? A possible role for prolactin. Musicae Scientiae, 15(2), 146–158.

35.

Izard

C. E.

(1992). Basic emotions, relations among emotions, and emotion-cognition relations. Psychological Review, 99(3), 561–565.

36.

Jack

R. E.

Garrod

O. G. B.

Caldara

Schyns

P. G.

(2012). Facial expressions of emotion are not culturally universal. Proceedings of the National Academy of Sciences of the United States of America, 109(19), 7241–7244.

37.

Johnson-Laird

P. N.

Oatley

(1989). The language of emotions: An analysis of a semantic field. Cognition and Emotion, 3(2), 81–123.

38.

Juslin

P. N.

(2011). Music and emotion: Seven questions, seven answers. In Deliège

Davidson

J. W.

(Eds.), Music and the mind: Essays in honour of John Sloboda (pp. 113–138). New York, NY: Oxford University Press.

39.

Juslin

P. N.

(2013). From everyday emotions to aesthetic emotions: Toward a unified theory of musical emotions. Physics of Life Reviews, 10, 235–266.

40.

Juslin

P. N.

Laukka

(2003). Communication of emotions in vocal expression and music performance: Different channels, same code? Psychological Bulletin, 129(5), 770–814.

41.

Juslin

P. N.

Västfjäll

(2008). Emotional responses to music: The need to consider underlying mechanisms. Behavioral and Brain Sciences, 31(5), 1–63.

42.

Juslin

P. N.

Harmat

Eerola

(2014). What makes music emotionally significant? Exploring the underlying mechanisms. Psychology of Music, 42(4), 599–623.

43.

Kallinen

(2004). Emotion related psychophysiological responses to listening music with eyes open versus eyes-closed: electrodermal (EDA), electrocardiac (ECG), and electromyographic (EMG) measures. In Lipscomb

S.D.

Ashley

Gjerdingen

R.O.

Webster

(Eds.), Proceedings of the 8th International Conference on Music Perception and Cognition (pp. 299–301). Rundle Mall, Australia: Causal Productions.

44.

Kawakami

Furukawa

Katahira

Okanoya

(2013). Sad music induces pleasant emotion. Frontiers in Psychology, 4, 311 (article 311).

45.

Kivy

(1990). Music alone: Philosophical reflections on the purely musical experience. Ithaca, NY: Cornell University Press.

46.

Konečni

V. J.

(2008). Does music induce emotion? A theoretical and methodological analysis. Psychology of Aesthetics, Creativity, and the Arts, 2(2), 115–129.

47.

Konečni

V. J.

Wanic

R. A.

Brown

(2007). Emotional and aesthetic antecedents and consequences of music-induced thrills. The American Journal of Psychology, 120(4), 619–643.

48.

Krumhuber

E. G.

Scherer

K. R.

(2011). Affect bursts: Dynamic patterns of facial expression. Emotion, 11(4), 825–841.

49.

Lang

P. J.

Greenwald

M. K.

Bradley

M. M.

Hamm

A. O.

(1993). Looking at pictures: Affective, facial, visceral, and behavioral reactions. Psychophysiology, 30(3), 261–273.

50.

Larsen

J. T.

McGraw

A. P.

(2011). Further evidence for mixed emotions. Journal of Personality and Social Psychology, 100(6), 1095–1110.

51.

Lench

H. C.

Flores

S. A.

Bench

S. W.

(2011). Discrete emotions predict changes in cognition, judgment, experience, behavior, and physiology: A meta-analysis of experimental emotion elicitations. Psychological Bulletin, 137(5), 834–855.

52.

Lundqvist

L. O.

Carlsson

Hilmersson

Juslin

P. N.

(2008). Emotional responses to music: Experience, expression, and physiology. Psychology of Music, 37(1), 61–90.

53.

Marsh

A. A.

Elfenbein

H. A.

Ambady

(2003). Nonverbal “accents”: Cultural differences in facial expressions of emotion. Psychological Science, 14(4), 373–376.

54.

Moriguchi

Ohnishi

Kawachi

Mori

Hirakata

Yamada

Komaki

(2005). Specific brain activation in Japanese and Caucasian people to fearful faces. Neuroreport, 16(2), 133–136.

55.

Muth

Raab

M. H.

Carbon

C.-C.

(2015). The stream of experience when watching artistic movies: Dynamic aesthetic effects revealed by the continuous evaluation procedure (CEP). Frontiers in Psychology, 6, 365.

56.

Nagel

Kopiez

Grewe

Altenmüller

(2007). EMuJoy: Software for continuous measurement of perceived emotions in music. Behavior Research Methods, 39(2), 283–290.

57.

Nelson

N. L.

Russell

J. A.

(2013). Universality revisited. Emotion Review, 5(1), 8–15.

58.

Noldus Information Technology. (2008). FaceReader reference manual (Version 5). Wageningen, The Netherlands: Author.

59.

North

A. C.

Hargreaves

D. J.

(1995). Subjective complexity, familiarity, and liking for popular music. Psychomusicology, 14, 77–93.

60.

Panksepp

(1995). The emotional sources of “chills” induced by music. Music Perception, 13(2), 171–207.

61.

Pereira

C. S.

Teixeira

Figueiredo

Xavier

Castro

S. L.

Brattico

(2011). Music and emotions in the brain: Familiarity matters. PLoS ONE, 6(11), e27241–9.

62.

Peretz

Gagnon

Bouchard

(1998). Music and emotion: Perceptual determinants, immediacy, and isolation after brain damage. Cognition, 68(2), 111–141.

63.

Russell

J. A.

(1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39, 1161–1178.

64.

Russell

J. A.

(1994). Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies. Psychological Bulletin, 115(1), 102–141.

65.

Salimpoor

V. N.

Benovoy

Larcher

Dagher

Zatorre

R. J.

(2011). Anatomically distinct dopamine release during anticipation and experience of peak emotion to music. Nature Neuroscience, 14(2), 257–262.

66.

Salimpoor

V. N.

Benovoy

Longo

Cooperstock

J. R.

Zatorre

R. J.

(2009). The rewarding aspects of music listening are related to degree of emotional arousal. PLoS ONE, 4(10), e7487.

67.

Scherer

K. R.

Coutinho

(2013). How music creates emotion: A multifactorial process approach. In Cochrane

Fantini

Scherer

K. R.

(Eds.), The emotional power of music: Multidisciplinary perspectives on musical expression, arousal, and social control (pp. 121–146). Oxford, UK: Oxford University Press.

68.

Schneider

Shiffrin

R. M.

(1977). Controlled and automatic human information processing: I. Detection, search, and attention. Psychological Review, 84(1), 1–66.

69.

Schubert

(1999). Measuring emotion continuously: Validity and reliability of the two-dimensional emotion-space. Australian Journal of Psychology, 51(3), 154–165.

70.

Schubert

(2010). Continuous self-report methods. In Juslin

P. N.

Sloboda

J. A.

(Eds.), Handbook of music and emotion: Theory, research, applications (pp. 223–253). Oxford, UK: Oxford University Press.

71.

Sedikides

Wildschut

Baden

(2004). Nostalgia: Conceptual issues and existential functions. In Greenberg

Koole

Pyszczynski

(Eds.), Handbook of experimental existential psychology (pp. 200–214). New York, NY: Guilford.

72.

Sloboda

J. A.

(1991). Music structure and emotional response: Some empirical findings. Psychology of Music, 19(2), 110–120.

73.

Taruffi

Koelsch

(2014). The paradox of music-evoked sadness: An online survey. PLoS ONE, 9(10), e110490.

74.

Thompson

W. F.

Schellenberg

E. G.

Husain

(2001). Arousal, mood, and the Mozart effect. Psychological Science, 12(3), 248–251.

75.

Van den Tol

A. J. M.

Edwards

(2013). Exploring a rationale for choosing to listen to sad music when feeling sad. Psychology of Music, 41(4), 440–465.

76.

Van den Tol

A. J. M.

Edwards

(2015). Listening to sad music in adverse situations: How music selection strategies relate to self-regulatory goals, listening effects, and mood enhancement. Psychology of Music, 43, 473–494.

77.

Van Kuilenburg

Wiering

den Uyl

(2005). A model based method for automatic facial expression recognition. In Gama

Camacho

Brazdil

Jorge

A. M.

Torgo

(Eds.), Proceedings of the 16th European Conference on Machine Learning (pp. 194–205). Berlin, Germany: Springer Verlag.

78.

Viola

Jones

(2001). Rapid object detection using a boosted cascade of simple features. Proceedings of the Computer Society Conference on Computer Vision and Pattern Recognition, 1, 511–518.

79.

Vuoskoski

J. K.

Eerola

(2012). Can sad music really make you sad? Indirect measures of affective states induced by music and autobiographical memories. Psychology of Aesthetics, Creativity, and the Arts, 6(3), 204–213.

80.

Weth

Kickinger

(2013, June). Ambivalent emotions in music: We like sad music when it makes us happy. Paper presented at the 3rd International Conference on Music & Emotion (ICME3), Jyväskylä, Finland. Retrieved from https://jyx.jyu.fi/dspace/handle/123456789/41597

81.

Weth

Parncutt

(2014, September). Functions of sad music in daily life: An explorative study on the influence of exposure, coping strategies and personality. Paper presented at the International Conference of Students of Systematic Musicology, London, UK. Retrieved from http://ojs.gold.ac.uk/index.php/sysmus14/article/view/261/276

82.

Whalen

P. J.

Rauch

S. L.

Etcoff

N. L.

McInerney

S. C.

Lee

M. B.

Jenike

M. A.

(1998). Masked presentations of emotional facial expressions modulate amygdala activity without explicit knowledge. The Journal of Neuroscience, 18(1), 411–418.

83.

Wildschut

Sedikides

Arndt

Routledge

(2006). Nostalgia: Content, triggers, functions. Journal of Personality and Social Psychology, 91(5), 975–993.

84.

Witvliet

C. V.

Vrana

S. R.

(1996). The emotional impact of instrumental music on affect ratings, facial EMG, autonomic measures, and the startle reflex: Effects of valence and arousal. Psychophysiology Supplement, 91.

85.

Zentner

Grandjean

Scherer

K. R.

(2008). Emotions evoked by the sound of music: Characterization, classification, and measurement. Emotion, 8(4), 494–521.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.47 MB