Reinforcement learning (RL) in the context of artificial agents is typically used to produce behavioral responses as a function of the reward obtained by interaction with the environment. When the problem consists of learning the shortest path to a goal, it is common to use reward functions yielding a fixed value after each decision, for example a positive value if the target location has been attained and a negative value at each intermediate step. However, this fixed strategy may be overly simplistic for agents to adapt to dynamic environments, in which resources may vary from time to time. By contrast, there is significant evidence that most living beings internally modulate reward value as a function of their context to expand their range of adaptivity. Inspired by the potential of this operation, we present a review of its underlying processes and we introduce a simplified formalization for artificial agents. The performance of this formalism is tested by monitoring the adaptation of an agent endowed with a model of motivated actor–critic, embedded with our formalization of value and constrained by physiological stability, to environments with different resource distribution. Our main result shows that the manner in which reward is internally processed as a function of the agent’s motivational state, strongly influences adaptivity of the behavioral cycles generated and the agent’s physiological stability.
AckleyD.LittmanM. (1991). Interactions Between Learning and Evolution. In LangtonC.TaylorC.FarmerJ. D.RasmussenS. (Eds.), Artificial Life II, SFI Studies in the Sciences of Complexity (Vol. X, pp. 487–509). Addison-Wesley, Menlo Park, CA.
2.
AhlgrenA.HalbergF. (1990). Cycles of Nature: an Introduction to Biological Rhythms. National Teachers Association.
3.
AlexanderG.DeLongM.PLS. (1990). Basal ganglia-thalamocortical circuits: parallel substrates for motor, oculomotor, prefrontal and limbic functions. Progress in Brain Research, 85: 119–146.
4.
AshbyW. (1965). Design for a Brain: the Origin of Adaptive Behaviour. Chapman & Hall, London.
5.
Avila-GarcíaO.CañameroL. (2002, August). A comparison of behaviour selection architectures using viability indicators. In Proceedings of International Workshop on Biologically-Inspired Robotics: the Legacy of W. Grey Walter.
6.
BaldassarreG. (2002). A modular neural-network model of the basal ganglia’s role in learning and selecting motor behaviours. Journal of Cognitive Systems Research, 3: 5–13.
7.
BalleineB. W.O’DohertyJ. P. (2010). Human and rodent homologies in action control: Corticostriatal determinants of goal-directed and habitual action. Neuropsychopharmacology, 35: 48–69.
8.
ButzM. V.ShirinovE.ReifK. L. (2010). Self-organizing sensorimotor maps plus internal motivations yield animal-like behavior. Adaptive Behavior, 18(3-4): 315–337.
9.
CañameroL. D. (1997). Modeling Motivations and Emotions as a Basis for Intelligent Behavior. In JohnsonW. L. (Ed.), Proceedings of the First International Symposium on Autonomous Agents (Agents’97) (pp. 148–155). New York, NY: ACM Press.
10.
CardinalR.ParkinsonJ.HallJ.EverittB. (2002). Emotion and motivation: the role of the amygdala, ventral striatum, and prefrontal cortex. Neuroscience and Behavioral Reviews, 26(3): 321–352.
11.
ChibV.RangelA.ShimojoS.O’DohertyJ. (2009). Evidence for a common representation of decision values for dissimilar goods in human ventromedial prefrontal cortex. Journal of Neuroscience, 29: 12315–12320.
12.
ConinxA.GuillotA.GirardB. (2008). Adaptive motivation in a biomimetic action selection mechanism‘. In 2nd French Conference on Computational Neuroscience (NeuroComp 2008) (pp. 158–162).
13.
ConoverK. L.ShizgalP. (1994). Differential effects of postingestive feedback on the reward value of sucrose and lateral hypothalamic stimulation in rats. Behavioral Neuroscience, 108(3), 559–572.
14.
ConoverK. L.WoodsideB.ShizgalP. (1994). Effects of sodium depletion on competition and summation between rewarding effects of salt and lateral hypothalamic stimulation in the rat. Behavioral Neuroscience, 108(3): 549–558.
15.
CosI.CañameroL.HayesG. M. (2010). Learning affordances of consummatory behaviors: Motivation-driven adaptive perception. Adaptive Behavior, 18(3-4): 285–314.
16.
DamasioA. R. (2000). The Feeling of What Happens: Body and Emotion in the Making of Consciousness. Harcourt Brace.
17.
DamoulasT.Cos-AguileraI.HayesG.TaylorT. (2005). Valency for adaptive homeostatic agents: Relating evolution and learning. In Proceedings of the 8th European Conference on Artificial Life (Ecal2005), Canterbury, UK. (Vol. 3630, pp. 936–945). Springer; Berlin/Heidelberg.
18.
DickinsonA.BalleineB. (2001). The Role of Learning in Motivation. In GallistelC. (Ed.), (Third ed., Vol. 3, chap. Learning, Motivation and Emotion). New York, NY: Wiley.
19.
DittrichK.KlauerK. C. (2011). Does ignoring lead to worse evaluations? a new explanation of the stimulus devaluation effect. Cognition and Emotion, 1: 1–16.
20.
EderA. B.RothermundK. (2008). When do motor behaviors (mis)match affective stimuli? an evaluative coding view of approach and avoidance reactions. Journal of Experimental Psychology, 137(2): 262–281.
21.
FreudS. (1940). An Outline of Psychoanalysis. Hogarth Press, London.
22.
GibsonJ. (1986). The Ecological Approach to Visual Perception (ErlbaumL. Y.; Hillsdale, Ed.). Hillsdale, N.J.; London.
23.
GirardB.TabareauN.PhamQ.BerthozA.SlotineJ.-J. (2008). Where neuroscience and dynamic system theory meet autonomous robotics: a contracting basal ganglia model for action selection. Neural Networks, 21(4): 628–641.
24.
GrabenhorstF.RollsE. T. (2011). Value, pleasure and choice in the ventral prefrontal cortex. Trends in Cognitive Sciences, 15(2): 56–67.
25.
GrabenhorstF.RollsE. T.ParisB. A. (2008). From affective value to decision-making in the prefrontal cortex. European Journal of Neuroscience, 28: 1930–1939.
26.
Guitart-MasipM.FuentemillaL.BachD. R.HuysQ. J.DayanP.DolanR. J. (2011). Action dominates valence in anticipatory representations in the human striatum and dopaminergic midbrain. The Journal of Neuroscience, 31(21): 7867–7875.
27.
GurneyK.PrescottT. J.RedgraveP. (1998, September). The basal ganglia viewed as an action selection device. In Proceedings of the Eighth International Conference on Artificial Neural Networks (pp. 1033–1038).
28.
HareT. A.O’DohertyJ.CamererC. F.SchultzW.RangelA. (2008). Dissociating the role of the orbitofrontal cortex and the striatum in the computation of goal values and prediction errors. Journal of Neuroscience, 28(22): 5623–5630.
29.
HazyT. E.FrankM. J.O’ReillyR. C. (2007). Towards an executive without a homunculus: computational models of the prefrontal cortex/basal ganglia system. Philosophical Transactions from the Royal Society. Series B., 113: 281–292.
30.
HindeR. A. (1960). Energy Models of Motivation. In Symposium of Society of Experimental Biology (Vol. 14, pp. 199–213). Company of Biologists on behalf of the Society for Experimental Biology, at Cambridge University Press.
31.
HindeR. A. (1971). Critique of Energy Models of Motivation. In BindraD.StewartJ. (Eds.), Motivation (pp. 36–48). Penguin.
32.
HoukJ. C.AdamsJ. L.BartoA. G. (1995). Models of Information Processing in the Basal Ganglia. In HoukJ. C.DavisJ. L.B. D. G. (Eds.), A Model of How the Basal Ganglia Generate and Use Neural Signals that Predict Reinforcement (2nd. Edition (1998) ed., pp. 249–270). MIT Press.
33.
HuangX.WengJ. (2002). Novelty and reinforcement learning in the value system of developmental robots. In 2nd International Workshop on Epigenetic Robotics (pp. 57–55).
34.
HullC. (1943). Principles of Behaviour: an Introduction to Behaviour Theory. D. Appleton-Century Company Inc., New York.
35.
JinX.CostaR. M. (2010). Start/stop signals emerge in nigrostriatal circuits during sequence learning. Nature, 466(22): 457–462.
36.
KableJ.GlimcherP. (2007). The neural correlates of subjective value during intertemporal choice. European Journal of Neuroscience, 10(12): 1625–1633.
37.
KennerleyS. W.BehrensT. E.WallisJ. D. (2011). Double dissociation of value computations in orbitofrontal and anterior cingulate neurons. Nature Neuroscience, 14(12): 1581–1589.
38.
KhamassiM.LachèzeL.GirardB.GuillotA. (2005). Actor-critic models of reinforcement learning in the basal ganglia: From natural to artificial rats. Adaptive Behavior, 13(2): 131–148.
39.
KonidarisG.BartoA. (2006). An Adaptive Robot Motivational System. In NolfiS.. (Eds.), Animals to Animats 9: Proceedings of the 9th International Conference on Simulation of Adaptive Behavior (pp. 346–356). Springer Verlag.
40.
KonidarisG.KuindersmaS.BartoA.GrupenR. (2010). Constructing skill trees for reinforcement learning agents from demonstration trajectories. Advances in Neural Information Processing Systems, 23: 2402–2410.
41.
KonidarisG. D.HayesG. M. (2005). An architecture for behavior-based reinforcement learning. Adaptive Behavior, 13(1): 5–32.
42.
LiJ.DawN. D. (2012). Signals in human striatum are appropriate for policy update rather than value prediction. Journal of Neuroscience, 31(14): 5504–5511.
43.
LorenzK. (1966). Evolution and Modification of Behaviour. Methuen & Co Ltd, London.
44.
MatarićM.BrooksR. (1990). Cambrian Intelligence: the Early History of the New AI. In BrooksR. (Ed.), (chap. Learning a Distributed Map Representation Based on Navigation Behaviors). The MIT Press.
45.
McClureS. M.DawN. D.MontagueP. R. (2003, August). A computational substrate for incentive salience. TRENDS in Neurosciences, 26(8).
46.
McDougallW. (1913). The sources and direction of psychophysical energy. American Journal of Insanity.
47.
McFarlandD. J.SiblyR. M. (1975). The behavioural final common path. Proceedings of the Royal Society of London, 270: 265–293.
48.
McFarlandD.SpierE. (1997). Basic cycles, utility and opportunism in self-sufficient robots. Robotics and Autonomous Systems, 20: 179–190.
49.
MerrickK. E. (2010). Modelling behavior cycles as a value system of developmental robots. Adaptive Behavior, 18: 237–257.
PfeiferR. (1996, September). Building fungus eaters: design principles of autonomous agents. In MaesP.MataricM. J.MeyerJ.-A.PollackJ.WilsonS.W. (Eds.), From Animals to Animats 4: Fourth International Conference on Simulation of Adaptive Behavior (p. 3-12). Cambridge, MA: The MIT Press/Bradford Books.
52.
QuilodranR.RothM.ProcykE. (2008). Behavioral shifts and action valuation in the anterior cingulate cortex. Neuron, 57(2): 314–325.
53.
RedgraveP.PrescottT.GurneyK. (1999). The basal ganglia: a vertebrate solution to the selection problem?Neuroscience, 89: 1009–1023.
RollsE. T. (2004). The functions of the orbitofrontal cortex. Brain and Cognition, 55: 11–29.
56.
RollsE. T. (2005). Emotion Explained. Oxford University Press.
57.
RollsE. T.GrabenhorstF. (2008). The orbitofrontal cortex and beyond: From affect to decision-making. Progress in Neurobiology, 86: 216–244.
58.
RumelhartD.HintonG.WilliamsR. (1986). Learning Internal Representations by Error Propagation. In RumelhartD.McClellandJ. (Eds.), Parallel Distributed Processing: Explorations in the Microstructure of Cognition (Vol. I). MIT Press.
59.
RushworthM. F.MarsR. B.SummerfieldC. (2009). General mechanisms for making decisions?Current Opinion in Neurobiology, 19: 75–83.
60.
SamejimaK.UedaY.DoyaK.KimuraM. (2005). Representation of action-specific reward values in the striatum. Science, 310: 1337–1340.
61.
SchaefferM.RotteM. (2007). Favorite brands as cultural objects modulate reward circuit. Neuroreport, 18(2): 141–145.
62.
SchultzW.ApicellaP.LjungbergT. (1993). Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. Journal of Neuroscience (13): 900–913.
63.
SchultzW.tremblayK.HollermanJ. (2000). Reward processing in primate orbitofrontal cortex and basal ganglia. Cerebral Cortex, 10: 272–741.
64.
SethA. K. (2000). Agent-based modelling and the environmental complexity thesis. In HallamB.FloreanoD.HallamJ.HayesG.MeyerJ. (Eds.), From Animals to Animats 7: Proceedings of the Seventh International Conference on the Simulation of Adaptive Behavior (pp. 13–24). Cambridge, MA: MIT Press.
65.
ShizgalP. (1997). Neural basis of utility estimation. Current Opinion in Neurobiology, 7: 198–208.
66.
SmithK. S.BerridgeK. C.AldridgeJ. W. (2011). Disentangling pleasure from incentive salience and learning signals in brain reward circuitry. PNAS, 108(27): 255–264.
67.
SuttonR.BartoA. (1981). Towards a modern theory of adaptive networks: Expectation and prediction. Psychological Review, 198(88): 135–170.
68.
SuttonR. S.BartoA. G. (1998). Reinforcement Learning (T. D. et al., Ed.). MIT Press.
69.
TanjiJ.HoshiE. (2001). Behavioural planning in the prefrontal cortex. Current Opinion in Neurobiology, 11: 164–170.
70.
TinbergenN. (1951). The study of Instinct. Oxford University Press.
71.
ToussaintM. (2003). Learning a world model and planning with a self-organizing, dynamic neural system. In Proceedings of NIPS’ 2003.
72.
TurnerR. S.DesmurgetM. (2010). Basal ganglia contributions to motor control: a vigour tutor. Current Opinion in Neurobiology, 20: 704–716.
73.
VelásquezJ. (1998). Modeling emotion-based decision-making. In Proceedings of the 1998 AAAI Fall Symposium Emotional and Intelligent: the Tangled Knot of Cognition, (pp. 164–169). Orlando, FL: AAAI Press.
74.
WallisJ. D. (2012). Cross-species studies of orbitofrontal cortex and value-based decision-making. Nature Neuroscience, 15(1): 13–19.
75.
WallisJ. D.MillerE. (2003, October). Neuronal activity in primate dorsolateral and orbital prefrontal cortex during performance of a reward preference task. European Journal of Neuroscience, 18(7): 2069–81.
76.
WilsonS.W. (1991). The animat path to AI. In MeyerJ.-A.WilsonS. (Eds.), From Animals to Animats: Proceedings of the First International Conference on Simulation of Adaptive Behavior. The MIT Press.
