In this paper, we investigate the use of emotional information in the learning process of autonomous agents. Inspired by four dimensions that are commonly postulated by appraisal theories of emotions, we construct a set of reward features to guide the learning process and behaviour of a reinforcement learning (RL) agent that inhabits an environment of which it has only limited perception. Much like what occurs in biological agents, each reward feature evaluates a particular aspect of the (history of) interaction of the agent history with the environment, thereby, in a sense, replicating some aspects of appraisal processes observed in humans and other animals. Our experiments in several foraging scenarios demonstrate that by optimising the relative contributions of each reward feature, the resulting “emotional” RL agents perform better than standard goal-oriented agents, particularly in consideration of their inherent perceptual limitations. Our results support the claim that biological evolutionary adaptive mechanisms such as emotions can provide crucial clues in creating robust, general-purpose reward mechanisms for autonomous artificial agents, thereby allowing them to overcome some of the challenges imposed by their inherent limitations.
AberdeenD. (2003). A (revised) survey of approximate methods for solving partially observable Markov decision processes. Eveleigh, Australia: National ICT Australia.
2.
AhnH.PicardR. (2006). Affective cognitive learning and decision making: The role of emotions. In TrapplR. (Ed.), Proceedings of the 18th European meeting on cybernetics and systems research (pp. 1–6). Vienna: Austrian Society for Cybernetic Studies.
3.
ArmonyJ.Servan-SchreiberD.CohenJ.LeDouxJ. (1997). Computational modeling of emotion: Explorations through the anatomy and physiology of fear conditioning. Trends in Cognitive Sciences, 1, 28–34.
4.
ArnoldM. (1960). Emotion and personality. New York, NY: Columbia University Press.
5.
AuerP.Cesa-BianchiN.FischerP. (2002). Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47, 235–256.
6.
BobrowG. (1964). Natural language input for a computer problem solving system. PhD Thesis, Massachusetts Institute of Technology, MA.
7.
BrafmanR. (2003). R-MAX: A general polynomial time algorithm for near-optimal reinforcement learning. Journal of Machine Learning Research, 3, 213–231.
8.
BratmanJ.SinghS.LewisR.SorgJ. (2012). Strong mitigation: Nesting search for good policies within search for good reward. In 11th international joint conference on autonomous agents and multiagent systems (pp. 407–414).
9.
BroekensD.KostersW.VerbeekF. (2007). On affect and self-adaptation: Potential benefits of valence-controlled action-selection. In MiraJ.ÁlvarezJ. (Eds.), Bio-inspired modeling of cognitive tasks. New York, NY: Springer.
10.
CañameroD. (1997). Modeling motivations and emotions as a basis for intelligent behavior. In Proceedings of the 1st international conference on autonomous agents (pp. 148–155).
11.
CardinalR.ParkinsonJ.HallJ.EverittB. (2002). Emotion and motivation: The role of the amygdala, ventral striatum, and prefrontal cortex. Neuroscience and Biobehavioral Reviews, 26, 321–352.
12.
CassandraA. (1998). Exact and approximate algorithms for partially observable Markov decision processes. PhD Thesis, Brown University, RI.
13.
DamasioA. (1994). Descartes’ error: Emotion, reason, and the human brain. G. P. Putnam’s Sons.
14.
DawkinsM. (2000). Animal minds and animal emotions. American Zoologist, 40, 883–888.
15.
El-NasrM.YenJ.IoergerT. (2000). FLAME: Fuzzy logic adaptive model of emotions. Autonomous Agents and Multiagent Systems, 3, 219–257.
16.
EllsworthP.SchererK. (2003). Appraisal processes in emotion. In DavidsonR.SchererK.GoldsmithH. (Eds.), Handbook of the affective sciences. Oxford, UK: Oxford University Press.
FrijdaN.MesquitaB. (1998). The analysis of emotions: Dimensions of variation. In MascoloM.GriffinS. (Eds.), What develops in emotional development? Emotions, personality, and psychotherapy. Plenum Press.
19.
GadanhoS. (2003). Learning behavior-selection by emotions and cognition in a multi-goal robot task. Journal of Machine Learning Research, 4, 385–412.
20.
HogewoningE.BroekensD.J.EggermontJ.BovenkampE. (2007). Strategies for affect-controlled action-selection in Soar-RL. Nature Inspired Problem-Solving Methods in Knowledge Engineering, 4528, 501–510.
21.
JaakkolaT.SinghS.JordanM. (1995). Reinforcement learning algorithm for partially observable Markov decision problems. In Advances in neural information systems7 (pp. 345–352). Cambridge, MA: MIT Press.
22.
JacobsE.BroekensJ.JonkerC. (2014). Emergent dynamics of joy, distress, hope and fear in reinforcement learning agents. In AAMAS workshop on adaptive learning agents.
23.
KaelblingL.LittmanM.CassandraA. (1998). Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101, 99–134.
24.
KaelblingL.LittmanM.MooreA. (1996). Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4, 237–285.
LazarusR. (2001). Relational meaning and discrete emotions. In SchererK.SchorrA.JohnstoneT. (Eds.), Appraisal processes in emotion: Theory, methods, research (pp. 37–67). Oxford, UK: Oxford University Press.
27.
LeDouxJ. (2000). Emotion circuits in the brain. Annual Review of Neuroscience, 23, 155–184.
28.
LeDouxJ. (2007). The amygdala. Current Biology, 17, 868–874.
29.
LeventhalH.SchererK. (1987). The relationship of emotion to cognition: A functional approach to a semantic controversy. Cognition & Emotion, 1, 3–28.
30.
LittmanM. (1994). Memoryless policies: Theoretical limitations and practical results. In Proceedings of the 3rd international conference on simulation of adaptive behavior – from animals to animats (pp. 238–245).
31.
LittmanM.CassandraA.KaelblingL. (1995). Learning policies for partially observable environments: Scaling up. In Proceedings of the 12th international conference on machine learning (pp. 362–370).
32.
LopesM.LangT.ToussaintM.OudeyerP.-Y. (2012). Exploration in model-based reinforcement learning by empirically estimating learning progress. In Advances in neural information systems (pp. 206–214). Cambridge, MA: MIT Press.
33.
MadaniO.HanksS.CondonA. (1999). On the undecidability of probabilistic planning and infinite-horizon partially observable Markov decision problems. In Proceedings of the 16th AAAI conference on artificial intelligence (pp. 541–548).
34.
MarinierR. P.LairdJ. E.LewisR. L. (2009, March). A computational unification of cognitive behavior and emotion. Cognitive Systems Research, 10, 48–69.
35.
MarsellaS.GratchJ.PettaP. (2010). Computational models of emotion. In SchererK.BanzigerT.RoeschE. (Eds.), Blueprint for affective computing (pp. 21–44). Oxford, UK: Oxford University Press.
36.
MataricM. (1994). Reward functions for accelerated learning. In Proceedings of the 11th international conference on machine learning (pp. 157–164).
37.
McCallumA. (1995). Instance-based utile distinctions for reinforcement learning with hidden state. In Proceedings of the 12th international conference on machine learning (pp. 387–395).
38.
MeloF.LopesM.FerreiraR. (2010). Analysis of inverse reinforcement learning with perturbed demonstrations. In Proceedings of the 19th European conference on artificial intelligence (pp. 349–354).
39.
MinskyM. (1986). The society of mind. Simon & Schuster.
40.
MooreA.AtkesonC. (1993). Prioritized sweeping: Reinforcement learning with less data and less real time. Machine Learning, 13, 103–130.
41.
Moulin-FrierC.OudeyerP. (2013). Exploration strategies in developmental robotics: a unified probabilistic framework. In Proceedings of the 3rd international joint conference on development and learning and epigenetic robotics (pp. 1–6).
42.
NeuG.SzepesváriC. (2009). Training parsers by inverse reinforcement learning. Machine Learning, 77, 303–337.
43.
NewellA.SimonH. (1956). The logic theory machine: A complex information processing system. RAND Corporation.
44.
NgA.HaradaD.RusselS. (1999). Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the 16th international conference on machine learning (pp. 278–287).
45.
NgA.RussellS. (2000). Algorithms for inverse reinforcement learning. In Proceedings of the 17th international conference on machine learning (pp. 663–670).
46.
NiekumS.BartoA.SpectorL. (2010). Genetic programming for reward function search. IEEE Transactions on Autonomous Mental Development, 2, 83–90.
47.
PicardR. (2000). Affective computing. Cambridge, MA: MIT Press.
RamachandranD.AmirE. (2007). Bayesian inverse reinforcement learning. In Proceedings of the 20th international joint conference on artificial intelligence (pp. 2586–2591).
50.
RandløvJ.AlstrømP. (1998). Learning to drive a bicycle using reinforcement learning and shaping. In Proceedings of the 15th international conference on machine learning.
51.
ReisenzeinR. (2009). Emotions as metarepresentational states of mind: Naturalizing the belief–desire theory of emotion. Cognitive Systems Research, 10, 6–20.
52.
RosemanI. (2001). A model of appraisal in the emotion system: Integrating theory, research, and applications. In SchererK.SchorrA.JohnstoneT. (Eds.), Appraisal processes in emotion: Theory, methods, research (pp. 68–91). Oxford, UK: Oxford University Press.
53.
RosemanI.SmithC. (2001). Appraisal theory: Overview, assumptions, varieties, controversies. In SchererK.SchorrA.JohnstoneT. (Eds.), Appraisal processes in emotion: Theory, methods, research (pp. 3–19). Oxford, UK: Oxford University Press.
54.
RyanR.DeciE. (2000). Intrinsic and extrinsic motivations: Classic definitions and new directions. Contemporary Educational Psychology, 25, 54–67.
55.
SalichsM.MalfazM. (2006). Using emotions on autonomous agents: The role of happiness, sadness and fear. In Proceedings of the annual convention on ambient intelligence and simulated behavior (pp. 157–164).
56.
SchererK. (2001). Appraisal considered as a process of multilevel sequential checking. In SchererK.SchorrA.JohnstoneT. (Eds.), Appraisal processes in emotion: Theory, methods, research (pp. 92–120). Oxford, UK: Oxford University Press.
57.
SequeiraP.MeloF.PaivaA. (2011). Emotion-based intrinsic motivation for reinforcement learning agents. In Proceedings of the 4th international conference on affective computing and intelligent interaction (pp. 326–336).
58.
SequeiraP.MeloF.S.PaivaA. (2014). Emergence of emotional appraisal signals in reinforcement learning agents. Autonomous Agents and Multiagent Systems.
59.
SequeiraP.MeloF.PradaR.PaivaA. (2011). Emerging social awareness: Exploring intrinsic motivation in multiagent learning. In Proceedings of the 1st international joint conference on development and learning and epigenetic robotics (pp. 1–6).
60.
SimonH. (1967). Motivational and emotional controls of cognition. Psychological Review, 74, 29–39.
61.
SinghS.JaakkolaT.JordanM. (1994). Learning without state-estimation in partially observable Markovian decision processes. In Proceedings of the 11th international conference on machine learning (pp. 284–292).
62.
SinghS.LewisR.BartoA. (2009). Where do rewards come from? In TaatgenN.van RijnH. (Eds.), Proceedings of the annual conference of the Cognitive Science Society (pp. 2601–2606).
63.
SinghS.LewisR.BartoA.SorgJ. (2010). Intrinsically motivated reinforcement learning: An evolutionary perspective. IEEE Transactions on Autonomous Mental Development, 2, 70–82.
64.
SmithC.KirbyL. (2009). Putting appraisal in context: Toward a relational model of appraisal and emotion. Cognition & Emotion, 23, 1352–1372.
65.
SorgJ.SinghS.LewisR. (2010a). Internal rewards mitigate agent boundedness. In Proceedings of the 27th international conference on machine learning (pp. 1007–1014).
66.
SorgJ.SinghS.LewisR. (2010b). Reward design via online gradient ascent. In Advances in neural information systems23 (pp. 1–9). Cambridge, MA: MIT Press.
67.
SuttonR.BartoA. (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press.
WiewioraE. (2003). Potential-based shaping and Q-value initialization are equivalent. Journal of Artificial Intelligence Research, 19, 205–208.
70.
WinogradT. (1971). Procedures as a representation for data in a computer program for understanding natural language. Cambridge, MA: MIT Press.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
