Modelling human emotion in interactive environments: Physiological ensemble and grounded approaches for synthetic agents

Abstract

With the rising research in emotionally believable agents, several advances in agent technology have been made, ranging from interactive virtual agents to emotional mechanism simulations and emotional agent architectures. However, creating an emotionally believable agent capable of emotional thought is still largely out of reach. It has been proposed that being able to accurately model human emotion would allow agents to mimic human behaviour while these models are studied to create more accurate theoretical models. In light of these challenges, we present a general method for human emotional state modelling in interactive environments. The proposed method employs a three-layered classification process to model the arousal and valence (i.e., hedonic) emotional components, based on four selected psychophysiological metrics. Additionally, we also developed a simplified version of our system for use in real-time systems and low-fidelity applications. The modelled emotional states by both approaches compared favourably with a manual approach following the current best practices reported in the literature while also improving on its predictive ability. The obtained results indicate we are able to accurately predict human emotional states, both in offline and online scenarios with varying levels of granularity; thus, providing a transversal method for modelling and reproducing human emotional profiles.

Keywords

Psychophysiology emotional state recognition emotional modelling

Get full access to this article

View all access options for this article.

References

[1]

P.A.

Nogueira,

Rodrigues,

Oliveira and

L.E.

Nacke, Guided emotional state regulation: Understanding and shaping players’ affective experiences in digital games, in: 9th Annual AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (AIIDE), 2013.

[2]

Ermi and

Mäyrä, Fundamental components of the gameplay experience: Analysing immersion, in: Digital Games Research Association Conference: Changing Views – Worlds in Play, 2005.

[3]

Cavazza,

Pizzi,

Charles,

Vogt and

André, Emotional input for character-based interactive storytelling, in: Proc. of the 8th International Conference on Autonomous Agents and Multiagent Systems-Volume 1, 2009, pp. 313–320.

[4]

Leite,

Pereira,

Mascarenhas,

Castellano,

Martinho,

Prada and

Paiva, Closing the loop: From affect recognition to empathic interaction, in: 3rd Int. Workshop on Affect Interaction in Natural Environments, 2010.

[5]

G.N.

Yannakakis and

Togelius, Experience-driven procedural content generation, Transactions on Affective Computing 2(3) (2011), 147–161.

[6]

J.A.

Russel, A circumplex model of affect, Journal of Personality and Social Psychology 39(6) (1980), 1161–1178.

[7]

Plutchik, A general psychoevolutionary theory of emotion Emotion: Theory, Research, and Experience 1(1) (1980), 3–33.

[8]

Chanel,

Kronegg,

Grandjean and

Pun, Emotion assessment: Arousal evaluation using EEG’s and peripheral physiological signals, in: Proc. Int. Workshop on Multimedia Content Representation, Classification and Security, 2006, pp. 530–537.

[9]

V.H.V.G.

Moreira, BioStories Geração de Conteúdos Multimédia Dinâmicos Mediante Informação Biométrica da Audiência, PhD Thesis, University of Porto, 2010.

10.

[10]

Mandryk and

Atkins, A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies, International Journal of Human-Computer Studies 65(4) (Apr. 2007), 329–347.

11.

[11]

Drachen,

L.E.

Nacke,

Yannakakis and

A.L.

Pedersen, Correlation between heart rate, electrodermal activity and player experience in first-person shooter games, in: Proc. of the 5th ACM SIGGRAPH Symposium on Video Games, 2010, pp. 49–54.

12.

[12]

Leon,

Clarke,

Callaghan and

Sepulveda, A user-independent real-time emotion recognition system for software agents in domestic environments, Engineering Applications of Artificial Intelligence 20(3) (Apr. 2007), 337–345.

13.

[13]

Brown,

Grundlehner and

Penders, Towards wireless emotional valence detection from EEG, in: Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society, Vol. 2011, Aug. 2011, pp. 2188–2191.

14.

[14]

Hazlett, Measuring emotional valence during interactive experiences: Boys at video game play, in: Proc. of the SIGCHI Conference on Human Factors in Computing Systems, 2006, pp. 1023–1026.

15.

[15]

Hazlett and

Benedek, Measuring emotional valence to understand the user’s experience of software, International Journal of Human-Computer Studies 65(4) (Apr. 2007), 306–314.

16.

[16]

Haag,

Goronzy,

Schaich and

Williams, Emotion recognition using bio-sensors: First steps towards an automatic system, in: Affective Dialogue Systems, 2004.

17.

[17]

Nasoz,

C.L.

Lisetti,

Alvarez and

Finkelstein, Emotion recognition from physiological signals for user modeling of affect, in: Proc. of the 3rd Workshop on Affective and Attitude User Modelling, 2003.

18.

[18]

Figueiredo and

Paiva, ‘I want to slay that dragon’ – influencing choice in interactive storytelling, in: Digital Interactive Storytelling, 2010.

19.

[19]

Pedersen,

Togelius and

G.N.

Yannakakis, Modeling player experience for content creation, Computational Intelligence and AI in Games 2(1) (2009), 121–133.

20.

[20]

P.J.

Lang,

M.M.

Bradley and

B.N.

Cuthbert, International affective picture system (IAPS), 2008.

21.

[21]

R.M.

Stern,

W.J.

Ray and

K.S.

Quigley, Psychophysiological Recording, 2nd edn, Oxford University Press, New York, 2001.

22.

[22]

Levillain,

J.O.

Orero,

Rifqi and

Bouchon-Meunier, Characterizing player’s experience from physiological signals using fuzzy decision trees, in: IEEE Symposium on Computational Intelligence and Games (CIG), 2010, pp. 75–82.

23.

[23]

Y.-H.

Yang, A regression approach to music emotion recognition, IEEE Transactions on Audio, Speech, and Language Processing 16(2) (2008), 448–457.

24.

[24]

Aalto,

Wallius,

Näätänen,

Hiltunen,

Metsähonkala,

Sipilä and

Karlsson, Regression analysis utilizing subjective evaluation of emotional experience in PET studies on emotions, Brain Res Brain Res Protoc. 15(3) (2005), 142–154.

25.

[25]

P.A.

Nogueira,

Rodrigues,

Oliveira and

L.E.

Nacke, A regression-based method for lightweight emotional state detection in interactive environments, in: XVI Portuguese Conference on Artificial Intelligence (EPIA), 2013.

26.

[26]

Egges,

Kshirsagar and

Magnenat-thalmann, A model for personality and emotion simulation, in: Knowledge-Based Intelligent Information & Engineering Systems (KES), 2003.

27.

[27]

Strobl,

Malley and

Tutz, An introduction to recursive partitioning, Psychological Methods 4(14) (2009), 323–348.

28.

[28]

D.A.

Táavara and

Meir, Agents with personality for videogames, in: AMDO’06 Proc. of the 4th International Conference on Articulated Motion and Deformable Objects, 2006, pp. 484–493.

29.

[29]

L.E.

Nacke, An introduction to physiological player metrics for evaluating games, in: Game Analytics, Springer, London, 2013, pp. 585–619.

30.

[30]

Rokach, Ensemble-based classifiers, Artificial Intelligence Review 33(1–2) (2005), 1–39.

31.

[31]

P.A.

Nogueira,

Rodrigues,

Oliveira and

L.E.

Nacke, A hybrid approach at emotional state detection: Merging theoretical models of emotion with data-driven statistical classifiers, in: ACM International Conference on Intelligent Agent Technology (IAT) 2013, pp. 253–260.

32.

[32]

Serra and

P.A.

Nogueira, Personality simulation in interactive agents through emotional biases, in: European Conference on Modelling and Simulation (ECMS), 2013, pp. 25–31.

33.

[33]

Pereira,

Oliveira,

Moreira and

Sarmento, Towards an architecture for emotional BDI agents, in: Proc. of 12th Portuguese Conference on Artificial Intelligence, 2005, pp. 40–47.

34.

[34]

Jiang,

J.M.

Vidal and

M.N.

Huhns, EBDI: An architecture for emotional agents, in: Proc. of the 6th International Joint Conference on Autonomous Agents and Multiagent Systems, 2007, p. 11.

35.

[35]

Oliveira and

Sarmento, Emotional valence-based mechanisms and agent personality, in: Advances in Artificial Intelligence, 2002, pp. 771–780.

36.

[36]

Masuch,

Hartman and

Schuster, Emotional agents for interactive environments, in: Fourth International Conference on Creating, Connecting and Collaborating Through Computing (C5’06), 2006, pp. 96–102.

37.

[37]

A.S.

Rao and

M.P.

Georgeff, Modeling rational agents within a BDI-architecture, in: Proc. of the 2nd International Conference on Principles of Knowledge Representation and Reasoning, 1991, pp. 473–484.

38.

[38]

M.Y.

Lim,

Dias,

Aylett and

Paiva, Creating adaptive affective autonomous NPCs, Autonomous Agents and Multiagent Systems 24(2) (Dec. 2010), 287–311.

39.

[39]

H.V.D.

Parunak and

Bisson, A model of emotions for situated agents, in: AAMAS ’06 Proc. of the Fifth International Joint Conference on Autonomous Agents and Multiagent Systems, Vol. 2006, 2006.

40.

[40]

W.S.N.

Reilly, Believable social and emotional agents, PhD Thesis, University of Birmingham, 1996.

41.

[41]

K.R.

Scherer, Unconscious processes in emotion: The bulk of the iceberg, in: The Unconscious Emotion,

Niedenthal,

Feldman-Barret and

Winkielman, eds, 2005.

42.

[42]

Slater,

Moreton and

Buckley, Emotional agents as software interfaces, in: Proc. of the 2008 International Workshop on Emotion in HCI – Designing for People, 2008, pp. 38–43.