Reinforcement learning, a general and universally useful framework for learning from experience, has been broadly recognized as a critically important concept for understanding and shaping adaptive behavior, both in ethology and in artificial intelligence. A key component in reinforcement learning is the reward function, which, according to an emerging consensus, should be intrinsic to the learning agent and a matter of appraisal rather than a simple reflection of external outcomes. We describe an approach to intrinsically motivated reinforcement learning that involves various aspects of happiness, operationalized as dynamic estimates of well-being. In four experiments, in which simulated agents learned to explore and forage in simulated environments, we show that agents whose reward function properly balances momentary (hedonic) and longer-term (eudaimonic) well-being outperform agents equipped with standard fitness-oriented reward functions. Our findings suggest that happiness-based features can be useful in developing robust, general-purpose reward mechanisms for intrinsically motivated autonomous agents.
AhnH.PicardR. (2006). Affective cognitive learning and decision making: The role of emotions. In Proceedings of the 18th European meeting on cybernetics and systems research. Vienna, Austria: Austrian Society for Cybernetic Studies.
2.
AkaishiR.KollingN.BrownJ.RushworthM. (2016). Neural mechanisms of credit assignment in a multicue environment. The Journal of Neuroscience, 36(4), 1096–1112.
3.
BaldassarreG.MirolliM. (2013). Intrinsically motivated learning in natural and artificial systems. Berlin: Springer.
4.
BartoA. (1995). Adaptive critics and the basal ganglia. In HoukJ. C.DavisJ.BeiserD. (Eds.), Models of information processing in the basal ganglia (pp. 215–232). Cambridge, MA: MIT Press.
5.
BartoA. (2013). Intrinsic motivation and reinforcement learning. In BaldassarreG.MirolliM. (Eds.), Intrinsically motivated learning in natural and artificial systems (pp. 16–47). Berlin: Springer.
6.
BartoA.SimsekO. (2005). Intrinsic motivation for reinforcement learning systems. In Proceedings of the thirteenth yale workshop on adaptive and learning systems (pp. 113–118). New Haven, CT: Yale University.
BerridgeK. (2003). Pleasures of the brain. Brain and Cognition, 52, 106–128.
9.
BratmanJ.SinghS.SorgJ.LewisR. (2012). Strong mitigation: Nesting search for good policies within search for good reward. In Proceedings of the 11th international conference on autonomous agents and multiagent systems-volume 1 (pp. 407–414). Valencia, Spain: International Foundation for Autonomous Agents and Multiagent Systems.
10.
BroekensD. (2007). Affect and learning: A computational analysis. Leiden: Faculty of Science, LIACS, Leiden University.
11.
CohenJ.BlumK. (2002). Reward and decision. Neuron, 36, 193–198.
12.
ColomboM. (2014). Deep and beautiful. The reward prediction error hypothesis of dopamine. Studies in History and Philosophy of Science Part C: Studies in History and Philosophy of Biological and Biomedical Sciences, 45, 57–67.
13.
DaswaniM.LeikeJ. (2015). A definition of happiness for reinforcement learning agents. In Artificial general intelligence (pp. 231–240). Berlin, Germany: Springer.
14.
DollB. B.SimonD. A.DawN. D. (2012). The ubiquity of model-based reinforcement learning. Current Opinion in Neurobiology, 22, 1075-1081.
15.
EdelmanS. (2012). The happiness of pursuit. New York, NY: Basic Books.
16.
EiserJ.FazioR.StaffordT.PrescottT. (2003). Connectionist simulation of attitude learning: Asymmetries in the acquisition of positive and negative evaluations. Personality and Social Psychology Bulletin, 29, 1221–1235.
GaoY.EdelmanS. (2016). Between pleasure and contentment: Evolutionary dynamics of some possible parameters of happiness. PLoS One, 11(5).
19.
GlimcherP. (2011). Understanding dopamine and reinforcement learning: The dopamine reward prediction error hypothesis. Proceedings of the National Academy of Sciences, 108, 15647–15654.
20.
GottliebJ.OudeyerP.-Y.LopesM.BaranesA. (2013). Information-seeking, curiosity, and attention: Computational and neural mechanisms. Trends in Cognitive Sciences, 17, 585–593.
21.
HendersonL.KnightT. (2012). Integrating the hedonic and eudaimonic perspectives to more comprehensively understand wellbeing and pathways to wellbeing. International Journal of Wellbeing, 2, 196–221.
22.
KakadeS.DayanP. (2002). Dopamine: Generalization and bonuses. Neural Networks the Official Journal of the International Neural Network Society, 15(4-6), 549-559.
23.
KamilA. C.SargentT. D. (1981). Foraging behavior: Ecological, ethological, and psychological approaches. New York: Garland.
24.
KaplanF.OudeyerP. (2004). Maximizing learning progress: An internal reward system for development. In IidaF.PfeiferR.SteelsL.KuniyoshiY. (Eds.), Embodied artificial intelligence (pp. 259–270). Berlin: Springer.
LeeD.SeoH.JungM. W. (2012). Neural basis of reinforcement learning and decision making. Annual Review of Neuroscience, 35, 287–308.
27.
LindquistK.WagerT.KoberH.Bliss-MoreauE.BarrettL. (2012). The brain basis of emotion: A meta-analytic review. Behavioral and Brain Sciences, 35, 121–143.
28.
MarsellaS.GratchJ.PettaP. (2010). Computational models of emotion. A Blueprint for Affective Computing-A sourcebook and manual (pp. 21–46). New York, NY: Oxford University Press.
29.
MeryF. (2013). Natural variation in learning and memory. Current Opinion in Neurobiology, 23, 52–56.
30.
MinskyM. (2006). The emotion machine: Commonsense thinking, artificial intelligence, and the future of the human mind. New York: Simon & Schuster.
31.
MirolliM.SantucciV.BaldassarreG. (2013). Phasic dopamine as a prediction error of intrinsic and extrinsic reinforcements driving both action acquisition and reward maximization: A simulated robotic study. Neural Networks the Official Journal of the International Neural Network Society, 39C, 40–51.
32.
MnihV.KavukcuogluK.SilverD.RusuA. A.VenessJ.BellemareM. G.. . . HassabisD. (2015). Human-level control through deep reinforcement learning. Nature, 518, 529–533.
33.
NelsonK. (2015). A bio-social-cultural approach to early cognitive development: entering the community of minds. In ScottR.KosslynS. (Eds.), Emerging trends in the social and behavioral sciences. New York: John Wiley & Sons.
34.
NivY. (2009). Reinforcement learning in the brain. Journal of Mathematical Psychology, 53, 139–154.
35.
OudeyerP.KaplanF. (2007). What is intrinsic motivation? A typology of computational approaches. Frontiers in Neurorobotics, 1, 6.
36.
OudeyerP.KaplanF.HafnerV. (2007). Intrinsic motivation systems for autonomous mental development. IEEE Transactions on Evolutionary Computation, 11, 265–286.
RedgraveP.VautrelleN.ReynoldsJ. N. J. (2011). Functional properties of the basal ganglia’s re-entrant loop architecture: Selection and reinforcement. Neuroscience, 198, 138-151.
39.
RogersC. R. (1963). Actualizing tendency in relation to m̈otivesänd to consciousness. In JonesM. R. (Ed.), Nebraska symposium on motivation (pp. 1–24). Oxford: University of Nebraska Press.
40.
RumbellT.BarndenJ.DenhamS.WennekersT. (2012). Emotions in autonomous agents: Comparative analysis of mechanisms and functions. Autonomous Agents and Multi-Agent Systems, 25, 1–45.
41.
RutledgeR.SkandaliN.DayanR.DolanR. (2014). A computational and neural model of momentary subjective well-being. Proceedings of the National Academy of Sciences, 111, 12252–12257.
42.
SalichsM. A.MalfazM. (2006). Using emotions on autonomous agents. The role of happiness, sadness and fear. Integrative Approaches to Machine Consciousness, Part of AISB, 6, 157–164.
43.
SalichsM. A.MalfazM. (2012). A new approach to modeling emotions and their use on a decision-making system for artificial agents. IEEE Transactions on Affective Computing, 3, 56–68.
44.
SchembriM.MirolliM.BaldassarreG. (2007). Evolving internal reinforcers for an intrinsically motivated reinforcement-learning robot. In IEEE international conference on development and learning (pp. 282–287). Imperial College, London: IEEE.
45.
SchererK. R.SchorrA.JohnstoneT. (2001). Appraisal processes in emotion. New York: Oxford University Press.
46.
SchmidhuberJ. (2015). Deep learning in neural networks: An overview. Neural Networks, 61, 85–117.
47.
SequeiraP.MeloF.PaivaA. (2011). Emotion-based intrinsic motivation for reinforcement learning agents. In Affective computing and intelligent interaction (pp. 326–336). Berlin: Springer.
48.
SequeiraP.PedroM.FranciscoS.PaivaA. (2014). Learning by appraising: An emotion-based approach to intrinsic reward design. Adaptive Behavior, 22, 330–349.
49.
SilverD.HuangA.MaddisonC. J.GuezA.SifreL.van den DriesscheG.. . . HassabisD. (2016). Mastering the game of Go with deep neural networks and tree search. Nature, 529, 484–503.
50.
SinghS.LewisR.BartoA. (2009). Where do rewards come from. In Proceedings of the annual conference of the cognitive science society (pp. 2601–2606). Amsterdam, NL: Cognitive Science Society.
51.
SinghS.LewisR.BartoA.SorgJ. (2010b). Intrinsically motivated reinforcement learning: An evolutionary perspective. IEEE Transactions on Autonomous Mental Development, 2, 70–82.
52.
SinghS.LewisR. L.BartoA. G.SorgJ. (2010a). Intrinsically motivated reinforcement learning: An evolutionary perspective. IEEE Transactions on Autonomous Mental Development, 2, 70-82.
53.
SorgJ.SinghS.LewisR. (2010). Internal rewards mitigate agent boundedness. In Proceedings of the 27th international conference on machine learning (pp. 1007–1014). Haifa, Israel: Omnipress.
54.
SuttonR.BartoA. (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT press.
55.
TapiaD. H.SilvaA. X.BallesterosG. I.FigueroaC. C.NiemeyerH. M.RamírezC. C. (2015). Differences in learning and memory of host plant features between specialist and generalist phytophagous insects. Animal Behaviour, 106, 1–10.
