Emotion and attention: Audiovisual models for group-level skin response recognition in short movies

Abstract

The electrodermal activity (EDA) is a psychophysiological indicator which can be considered a somatic marker of the emotional and attentional reaction of subjects towards stimuli. EDA measurements are not biased by the cognitive process of giving an opinion or a score to characterize the subjective perception, and group-level EDA recordings integrate the reaction of the whole audience, thus reducing the signal noise. This paper contributes to the field of affective video content analysis, extending previous novel work on the use of EDA as ground truth for prediction algorithms. Here, we label short video clips according to the audience’s emotion (high vs. low) and attention (increasing vs. decreasing), derived from EDA records. Then, we propose a set of low-level audiovisual descriptors and train binary classifiers that predict the emotion and attention with 75% and 80% accuracy, respectively. These results, along with those of previous works, reinforce the usefulness of such low-level audiovisual descriptors to model video in terms of the induced affective response.

Keywords

Electrodermal activity emotion attention affective video content analysis audiovisual descriptors

Get full access to this article

View all access options for this article.

References

Aiger,

Palacín and

J.-M.

Cornejo, Electrodermal signal by sociograph: Methodology to measure the group activity, Revista de Psicología Social28(3) (2013), 333–347. doi:10.1174/021347413807719102.

Asahina,

Suzuki,

Mori,

Kanesaka and

Hattori, Emotional sweating response in a patient with bilateral amygdala damage, International Journal of Psychophysiology47(1) (2003), 87–93. doi:10.1016/S0167-8760(02)00123-X. http://www.sciencedirect.com/science/article/pii/S016787600200123X .

Augoustinos,

Walker and

Donaghue, Social Cognition: An Integrated Introduction, Sage, 2014, pp. 114–115.

Ayata,

Yaslan and

Kamaşak, Emotion recognition via random forest and galvanic skin response: Comparison of time based feature sets, window sizes and wavelet approaches, 2016, pp. 1–4. doi:10.1109/TIPTEKNO.2016.7863130.

Bernardi,

Porta and

Sleight, Cardiovascular, cerebrovascular, and respiratory changes induced by different types of music in musicians and non-musicians: The importance of silence, Heart92(4) (2006), 445–452. doi:10.1136/hrt.2005.064600.

Bordwell,

Thompson and

Ashton, Film Art: An Introduction, Vol. 7, McGraw-Hill, New York, 1997.

Boucsein, Electrodermal Activity, Springer Science & Business Media, 2012. doi:10.1007/978-1-4614-1126-0.

S.J.

Breckler, Empirical validation of affect, behavior, and cognition as distinct components of attitude, Journal of Personality and Social Psychology47(6) (1984), 1191–1205. doi:10.1037/0022-3514.47.6.1191.

Brinckmann, Color and Empathy: Essays on Two Aspects of Film, Amsterdam University Press, 2014. doi:10.1515/9789048523269.

10.

Cabeza and

Nyberg, Imaging cognition II: An empirical review of 275 PET and fMRI studies, J. Cognitive Neuroscience12(1) (2000), 1–47. doi:10.1162/08989290051137585.

11.

A.J.

Cohen, Music as a source of emotion in film, in: Music and Emotion: Theory and Research, 2001, pp. 249–272.

12.

A.R.

Damasio, Descartes’ Error: Emotion, Reason, and the Human Brain, Avon Books, New York, 1994.

13.

Darwin, The Expression of the Emotions in Man and Animals, John Marry, London, UK, 1872.

14.

Datta,

Joshi,

Li and

J.Z.

Wang, Studying aesthetics in photographic images using a computational approach, in: Proceedings of the 9th European Conference on Computer Vision – Volume Part III, ECCV’06, Springer-Verlag, Berlin, Heidelberg, 2006, pp. 288–301. ISBN 3-540-33836-5, 978-3-540-33836-9. doi:10.1007/11744078_23.

15.

M.E.

Dawson and

K.H.

Nuechterlein, Psychophysiological dysfunctions in the developmental course of schizophrenic disorders, Schizophrenia Bulletin10(2) (1984), 204–232. doi:10.1093/schbul/10.2.204. http://schizophreniabulletin.oxfordjournals.org/content/10/2/204.abstract .

16.

M.E.

Dawson,

A.M.

Schell and

D.L.

Filion, The electrodermal system, in: Handbook of Psychophysiology,

J.T.

Cacioppo,

L.G.

Tassinary and

G.G.

Berntson, eds, 2nd edn, Cambridge University Press, Cambridge, 2000, pp. 200–223.

17.

R.T.

Dean,

Bailes and

Schubert, Acoustic intensity causes perceived changes in arousal levels in music: An experimental investigation, PloS One6(4) (2011), e18591.

18.

Féré, Note sur les modifications de la résistance électrique sous l’influence des excitations sensorielles et des émotions, CR Soc. Biol5 (1888), 217–219.

19.

Fernández-Martínez,

Hernández-García and

Díaz-de-María, Succeeding metadata based annotation scheme and visual tips for the automatic assessment of video aesthetic quality in car commercials, Expert Systems with Applications (2015), 293–305. doi:10.1016/j.eswa.2014.07.033.

20.

Fernández-Martínez,

Hernández-García,

Gallardo-Antolín and

F.D.

de María, Combining audio-visual features for viewers’ perception classification of Youtube car commercials, in: Proceedings of Workshop on Speech, Language and Audio in Multimedia (SLAM), 2014.

21.

Fleureau,

Guillotel and

Orlac, Affective benchmarking of movies based on the physiological responses of a real audience, in: Affective Computing and Intelligent Interaction (ACII), 2013 Humaine Association Conference on, 2013, pp. 73–78, doi:10.1109/ACII.2013.19. ISSN 2156-8103.

22.

D.C.

Fowles,

M.J.

Christie,

Edelberg,

W.W.

Grings,

D.T.

Lykken and

P.H.

Venables, Publication recommendations for electrodermal measurements, Psychophysiology18(3) (1981), 232–239. doi:10.1111/j.1469-8986.1981.tb03024.x.

23.

Gagnon and

Peretz, Mode and tempo relative contributions to “happy-sad” judgements in equitone melodies, Cognition and emotion17(1) (2003), 25–40. doi:10.1080/02699930302279.

24.

Gendron and

L.F.

Barrett, Reconstructing the past: A century of ideas about emotion in psychology, Emotion review1(4) (2009), 316–339. doi:10.1177/1754073909338877.

25.

Guyon,

Weston,

Barnhill and

Vapnik, Gene selection for cancer classification using support vector machines, Mach. Learn.46(1–3) (2002), 389–422. doi:10.1023/A:1012487302797.

26.

J.C.

Hailstone,

Omar,

S.M.

Henley,

Frost,

M.G.

Kenward and

J.D.

Warren, It’s not what you play, it’s how you play it: Timbre affects perception of emotion in music, The quarterly Journal of Experimental psychology62(11) (2009), 2141–2155. doi:10.1080/17470210902765957.

27.

Hall,

Frank,

Holmes,

Pfahringer,

Reutemann and

I.H.

Witten, The WEKA data mining software: An update, ACM SIGKDD explorations newsletter11(1) (2009), 10–18. doi:10.1145/1656274.1656278.

28.

Hernández-García,

Fernández-Martínez and

Díaz-de-María, Comparing visual descriptors and automatic rating strategies for video aesthetics prediction, Signal Processing: Image Communication47 (2016), 280–288.

29.

Hernández-García,

Fernández-Martínez and

Díaz-de-María, Emotion and attention: Predicting electrodermal activity through video visual descriptors, in: Proceedings of the International Conference on Web Intelligence, ACM, 2017, pp. 914–923. doi:10.1145/3106426.3109418.

30.

E.R.

Hilgard, The trilogy of mind: Cognition, affection, and conation, Journal of the History of the Behavioral Sciences16(2) (1980), 107–117. doi:10.1002/1520-6696(198004)16:2<107::AID-JHBS2300160202>3.0.CO;2-Y.

31.

James, What is an emotion?, Mind34 (1884), 188–205. doi:10.1093/mind/os-IX.34.188.

32.

James, The Principles of Psychology, Dover, New York, 1890.

33.

Koelstra,

Muhl,

Soleymani,

J.-S.

Lee,

Yazdani,

Ebrahimi,

Pun,

Nijholt and

Patras, DEAP: A database for emotion analysis using physiological signals, IEEE Transactions on Affective Computing3(1) (2012), 18–31. doi:10.1109/T-AFFC.2011.15.

34.

Lajante,

Droulers,

Dondaine and

Amarantini, Opening the “black box” of electrodermal activity in consumer neuroscience research., Journal of Neuroscience, Psychology, and Economics5(4) (2012), 238. doi:10.1037/a0030680.

35.

P.J.

Lang,

M.K.

Greenwald,

M.M.

Bradley and

A.O.

Hamm, Looking at pictures: Affective, facial, visceral, and behavioral reactions, Psychophysiology30(3) (1993), 261–273. doi:10.1111/j.1469-8986.1993.tb03352.x.

36.

Latrillot and

Toiviainen, MIR in Matlab: A toolbox for musical feature extraction, in: Proceedings of the International Conference on Music Information Retrieval, 2007.

37.

LeDoux, Rethinking the emotional brain, Neuron73(4) (2012), 653–676. doi:10.1016/j.neuron.2012.02.004.

38.

J.L.

Martínez and

Garrido, Sistema para la medición de reacciones emocionales en grupos sociales. 2 168 928, 2003.

39.

J.L.

Martínez,

Monge and

M.I.

Valdunquillo, Medición de las respuestas psicofisiológicas grupales para apoyar el análisis de discursos políticos, Trípodos29 (2012), 53–72.

40.

A.K.

Moorthy,

Obrador and

Oliver, Towards computational models of the visual aesthetic appeal of consumer videos, in: Proceedings of the 11th European Conference on Computer Vision: Part V, ECCV’10, Springer-Verlag, Berlin, Heidelberg, 2010, pp. 1–14. ISBN 3-642-15554-5, 978-3-642-15554-3. http://dl.acm.org/citation.cfm?id=1888150.1888152 .

41.

Öhman, Distinguishing unconscious from conscious emotional processes: Methodological considerations and theoretical implications, in: Handbook of Cognition and Emotion, John Wiley & Sons, Ltd, 1999, pp. 321–352. ISBN 9780470013496. doi:10.1002/0470013494.ch17.

42.

Ondaatje and

Murch, The Conversations: Walter Murch and the Art of Editing Film, A&C Black, 2002.

43.

Pessoa, On the relationship between emotion and cognition, Nature Reviews Neuroscience9(2) (2008), 148–158. doi:10.1038/nrn2317.

44.

K.L.

Phan,

Wager,

S.F.

Taylor and

Liberzon, Functional neuroanatomy of emotion: A meta-analysis of emotion activation studies in PET and fMRI, Neuroimage16(2) (2002), 331–348. doi:10.1006/nimg.2002.1087.

45.

M.I.

Posner,

M.R.

Rueda and

Kanske, Probing the mechanisms of attention, in: Handbook of Psychophysiology,

J.T.

Cacioppo,

L.G.

Tassinary and

Berntson, eds, 3rd edn, Cambridge University Press, 2007, pp. 410–432, Cambridge Books Online. ISBN 9780511546396. doi:10.1017/CBO9780511546396.018.

46.

W.F.

Prokasy and

D.C.

Raskin (eds), Electrodermal Activity in Psychological Research, Academic Press, 1973. ISBN 978-0-12-565950-5. doi:10.1016/B978-0-12-565950-5.50001-6.

47.

Ronan,

Sazdov and

Ward, Loudness normalisation: Paradigm shift or placebo for the use of hyper-compression in pop music?, in: International Computer Music Conference, 2014.

48.

K.R.

Scherer, What are emotions? And how can they be measured?, Social science information44(4) (2005), 695–729. doi:10.1177/0539018405058216.

49.

Schubert, Modeling perceived emotion with continuous musical features, Music Perception: An Interdisciplinary Journal21(4) (2004), 561–585. doi:10.1525/mp.2004.21.4.561.

50.

Sequeira,

Hot,

Silvert and

Delplanque, Electrical autonomic correlates of emotion, International journal of psychophysiology71(1) (2009), 50–56. doi:10.1016/j.ijpsycho.2008.07.009.

51.

Soleymani,

Chanel,

J.J.M.

Kierkels and

Pun, Affective characterization of movie scenes based on multimedia content analysis and user’s physiological emotional responses., in: International Symposium on Multimedia, IEEE Computer Society, 2008, pp. 228–235, ISBN 978-0-7695-3454-1. doi:10.1109/ISM.2008.14. http://dblp.uni-trier.de/db/conf/ism/ism2008.html#SoleymaniCKP08 .

52.

Soleymani,

Lichtenauer,

Pun and

Pantic, A multimodal database for affect recognition and implicit tagging, IEEE Transactions on Affective Computing3(1) (2012), 42–55. doi:10.1109/T-AFFC.2011.25.

53.

K.L.

Subotnik,

A.M.

Schell,

M.S.

Chilingar,

M.E.

Dawson,

Ventura,

K.A.

Kelly,

G.S.

Hellemann and

K.H.

Nuechterlein, The interaction of electrodermal activity and expressed emotion in predicting symptoms in recent-onset schizophrenia, Psychophysiology49(8) (2012), 1035–1038. doi:10.1111/j.1469-8986.2012.01383.x.

54.

E.S.

Tan (ed.), Emotion and the Structure of Narrative Film: Film as an Emotion Machine, Lawrence Erlbaum Associates, Inc., 1996.

55.

Tranel, Electrodermal activity in cognitive neuroscience: Neuroanatomical and neuropsychological correlates, in: Cognitive Neuroscience of Emotion. Series in Affective Science, Oxford University Press, 2000, pp. 192–224.

56.

Y.J.

Wang and

M.S.

Minor, Validity, reliability, and applicability of psychophysiological techniques in marketing research, Psychology & Marketing25(2) (2008), 197–232. doi:10.1002/mar.20206.

57.

R.B.

Zajonc, Feeling and thinking: Preferences need no inferences., American Psychologist35(2) (1980), 151–175. doi:10.1037/0003-066x.35.2.151.