Biological organisms have the ability to respond quickly to an ever-changing world. Because this adaptability is so critical for survival, all vertebrates have sub-cortical structures, which comprise the neuromodulatory systems, to regulate fundamental behavior and drive decision making in response to environmental events. In the vertebrate, there are separate neuromodulators that respond to threats, reward anticipation, novelty, and attentional effort. However, each of these neuromodulatory systems has a similar effect, that is, to cause an organism to be decisive when environmental conditions call for such actions, and allow the organism to be more exploratory when there are no pressing events. In this article, it is proposed that principles of the neuromodulatory system could provide a framework for controlling artificial agents that may improve current artificial agent behavior. These agents would operate autonomously, effectively explore their environment, and be decisive when environmental conditions call for action.
Alexander, W.H., & Sporns, O. (2002). An embodied model of learning, plasticity, and reward. Adaptive Behavior, 10, 143-159.
2.
Amat, J., Paul, E., Zarza, C., Watkins, L.R., & Maier, S.F. (2006). Previous experience with behavioral control over stress blocks the behavioral and dorsal raphe nucleus activating effects of later uncontrollable stress: role of the ventral medial prefrontal cortex . Journal of Neuroscience, 26, 13264-13272.
3.
Arleo, A., Smeraldi, F., & Gerstner, W. (2004). Cognitive navigation based on nonuniform Gabor space sampling, unsupervised growing networks, and reinforcement learning . IEEE Transactions on Neural Networks, 15, 639-652.
4.
Aston-Jones, G., & Cohen, J.D. (2005). Adaptive gain and the role of the locus coeruleus-norepinephrine system in optimal performance. Journal of Comparative Neurology , 493, 99-110.
5.
Avila-Garcia, O., & CaƱamero, L. (2004). Using hormonal feedback to modulate action selection in a competitive scenario. In From Animals to Animats 8, Proceedings of the 8th International Conference on Simulation of Adaptive Behavior (SAB'04) (pp. 243-252). Cambridge, MA: MIT Press.
6.
Barnes, N.M., & Sharp, T. (1999). A review of central 5-HT receptors and their function . Neuropharmacology, 38, 1083-1152.
7.
Baxter, M.G., & Chiba, A.A. (1999). Cognitive functions of the basal forebrain. Current Opinion in Neurobiology, 9, 178-183.
8.
Berridge, C.W., & Waterhouse, B.D. (2003). The locus coeruleus-noradrenergic system: modulation of behavioral state and state-dependent cognitive processes . Brain Research Reviews, 42, 33-84.
Blanchard, A., & CaƱamero, L. (2006). Modulation of exploratory behavior for adaptation to the context. In Proceedings of the AISB'06 Symposium on Biologically Inspired Robotics (Biro-net) (pp. 131-137). Bristol, UK: AISB Press.
11.
Bogacz, R., Usher, M., Zhang, J., & McClelland, J.L. (2007). Extending a biologically inspired model of choice: multi-alternatives, nonlinearity and value-based multidimensional choice. Philosophical Transactions of the Royal Society of London, 362, 1655-1670.
12.
Briand, L.A., Gritton, H., Howe, W.M., Young, D.A., & Sarter, M. (2007). Modulators in concert for cognition: Modulator interactions in the prefrontal cortex. Progress in Neurobiology , 83, 69-91.
13.
Broekens, J., Kosters, W.A., & Verbeek, F.J. (2007). Affect, anticipation, and adaptation: Affect-controlled selection of anticipatory simulation in artificial adaptive agents. Adaptive Behavior, 15, 397-422.
14.
Bucci, D.J., Holland, P.C., & Gallagher, M. (1998). Removal of cholinergic input to rat posterior parietal cortex disrupts incremental processing of conditioned stimuli. Journal of Neuroscience, 18, 8038-8046.
15.
CaƱamero, L.D. (1997). Modeling motivations and emotions as a basis for intelligent behavior. In Proceedings of the First International Conference on Autonomous Agents (pp. 148-155). New York: ACM Press.
16.
Chiba, A.A., Bucci, D.J., Holland, P.C., & Gallagher, M. (1995). Basal forebrain cholinergic lesions disrupt increments but not decrements in conditioned stimulus processing. Journal of Neuroscience, 15, 7315-7322.
17.
Cho, A. (2007). Robotics. Robotic cars tackle crosstown traffic-and not one another. Science, 318, 1060-1061.
18.
Damasio, A. (2005). Descartes' error: Emotion, reason, and the human brain. New York: Penguin.
19.
Doya, K. (2002). Metalearning and neuromodulation. Neural Networks, 15, 495-506.
20.
Doya, K. (2008). Modulators of decision making. Nature Neuroscience, 11, 410-416.
21.
Doya, K., & Uchibe, E. (2005). The Cyber Rodent project: Exploration of adaptive mechanisms for self-preservation and self-reproduction. Adaptive Behavior, 13, 149- 160.
22.
Fluxe, K., Hokfelt, T., Johansson, O., Jonsson, G., Lidbrink, P., & Ljungdahl, A. (1974). The origin of the dopamine nerve terminals in limbic and frontal cortex. Evidence for mesocortico dopamine neurons. Brain Research, 82, 349-355.
23.
Frank, M.J., & Claus, E.D. (2006). Anatomy of a decision: striato-orbitofrontal interactions in reinforcement learning, decision making, and reversal. Psychological Review, 113, 300-326.
24.
Frank, M.J., Santamaria, A., O'Reilly, R.C., & Willcutt, E. (2007). Testing computational models of dopamine and noradrenaline dysfunction in attention deficit/hyperactivity disorder. Neuropsychopharmacology, 32, 1583-1599.
25.
Frank, M.J., Seeberger, L.C., & O'Reilly, R.C. (2004). By carrot or by stick: cognitive reinforcement learning in parkinsonism. Science , 306, 1940-1943.
26.
Gibbs, W.W. (2004). From finish to start. Was the grand challenge robot race in March the fiasco it appeared to be? Hardly, argues William "Red" Whittaker. The annual event is pushing mobile robotics to get real. Scientific American, 291, 33-34.
27.
Gu, Q. (2002). Neuromodulatory transmitter systems in the cortex and their role in cortical plasticity. Neuroscience, 111, 815-835.
28.
Guenter, F., Hersch, M., Calinon, S., & Billard, A. (2007). Reinforcement learning for imitating constrained reaching movements. Advanced Robotics, 21, 1521-1544.
29.
Gurney, K., Prescott, T.J., & Redgrave, P. (2001a). A computational model of action selection in the basal ganglia. I. A new functional anatomy. Biological Cybernetics, 84, 401- 410.
30.
Gurney, K., Prescott, T.J., & Redgrave, P. (2001b). A computational model of action selection in the basal ganglia. II. Analysis and simulation of behaviour. Biological Cybernetics, 84, 411-423.
31.
Gurney, K., Prescott, T.J., Wickens, J.R., & Redgrave, P. (2004). Computational models of the basal ganglia: from robots to membranes. Trends in Neurosciences, 27, 453- 459.
32.
Han, J.S., Holland, P.C., & Gallagher, M. (1999). Disconnection of the amygdala central nucleus and substantia innominata/nucleus basalis disrupts increments in conditioned stimulus processing in rats. Behavioral Neuroscience, 113, 143-151.
33.
Harvey, J.A. (2003). Role of the serotonin 5-HT(2A) receptor in learning . Learning & Memory, 10, 355-362.
34.
Hasselmo, M.E., & McGaughy, J. (2004). High acetylcholine levels set circuit dynamics for attention and encoding and low acetylcholine levels set dynamics for consolidation . Progress in Brain Research, 145, 207-231.
35.
Humphries, M.D., Gurney, K., & Prescott, T.J. (2007). Is there a brainstem substrate for action selection? Philosophical Transactions of the Royal Society of London, 362, 1627-1639.
36.
Hyman, S.E., Malenka, R.C., & Nestler, E.J. (2006). Neural mechanisms of addiction: the role of reward-related learning and memory. Annual Review of Neuroscience, 29, 565-598.
37.
Iida, S., Kuwayama, K., Kanoh, M., Kato, S., & Itoh, H. (2004). A dynamic allocation method of basis functions in reinforcement learning. In Proceedings of AI 2004, Advances in Artificial Intelligence, Lecture notes in computer science (Vol. 3339, pp. 272-283). Berlin: Springer.
38.
Jones, J., & Roth, D. (2003). Robot programming : A practical guide to behavior-based robotics. New York: McGraw-Hill .
39.
Kobayashi, M., Imamura, K., Sugai, T., Onoda, N., Yamamoto, M., Komai, S., & Watanabe, Y. (2000). Selective suppression of horizontal propagation in rat visual cortex by norepinephrine. European Journal of Neuroscience, 12, 264- 272.
40.
Kondo, T., & Ito, K. (2004). A reinforcement learning with evolutionary state recruitment strategy for autonomous mobile robots control. Robotics and Autonomous Systems, 46, 111-124.
41.
Krichmar, J.L., & Edelman, G.M. (2002). Machine psychology: Autonomous behavior, perceptual categorization, and conditioning in a brain-based device. Cerebral Cortex, 12, 818-830.
42.
Krichmar, J.L., Seth, A.K., Nitz, D.A., Fleischer, J.G., & Edelman, G.M. (2005). Spatial navigation and causal analysis in a brain-based device modeling cortical-hippocampal interactions. Neuroinformatics , 3, 197-222.
43.
Lapish, C.C., Kroener, S., Durstewitz, D., Lavin, A., & Seamans, J.K. (2007). The ability of the mesocortical dopamine system to operate in distinct temporal modes. Psychopharmacology , 191, 609-625.
44.
Likhachev, M., & Arkin, R. (2000). Robotic comfort zones. In Proceedings of SPIE: Sensor Fusion and Decentralized Control in Robotic Systems III (pp. 27-41).
45.
McClure, S.M., Daw, N.D., & Montague, P.R. (2003). A computational substrate for incentive salience. Trends in Neurosciences, 26, 423-428.
46.
McGaugh, J.L. (2004). The amygdala modulates the consolidation of memories of emotionally arousing experiences. Annual Review of Neuroscience , 27, 1-28.
47.
McMahon, A., Scott, D., Baxter, P., & Browne, W. (2006). An autonomous explore/exploit strategy. In AISB'06 Symposium on Nature Inspired Systems (pp. 192-201). Bristol, UK: AISB Press.
48.
Meneses, A., & Perez-Garcia, G. (2007). 5-HT(1A) receptors and memory. Neuroscience & Biobehavioral Reviews, 31, 705-727.
49.
Millan, M.J. (2003). The neurobiology and control of anxious states . Progress in Neurobiology, 70, 83-244.
50.
Moffat, D., Phaf, R.H., & Frijda, N.H. (1993). Analysis of a model of emotions. In Prospects for Artificial Intelligence (pp. 219-228). Oxford: IOS Press.
51.
Nakamura, Y., Mori, T., Sato, M.A., & Ishii, S. (2007). Reinforcement learning for a biped robot based on a CPGactor-critic method. Neural Networks, 20, 723-735.
52.
Pauli, W.M., & O'Reilly, R.C. (2008). Attentional control of associative learning-A possible role of the central cholinergic system. Brain Research, 1202, 43-53.
53.
Pfeifer, R. (1994). The "fungus eater approach" to emotion: a view from artificial intelligence. Cognitive Studies: Bulletin of the Japanese Cognitive Science Society, 1, 42-57.
54.
Prescott, T.J., Montes Gonzalez, F.M., Gurney, K., Humphries, M.D., & Redgrave, P. (2006). A robot model of the basal ganglia: behavior and intrinsic processing. Neural Networks, 19, 31-61.
55.
Redgrave, P., & Gurney, K. (2006). The short-latency dopamine signal: a role in discovering novel actions? Nature Reviews Neuroscience, 7, 967-975.
56.
Robbins, T.W., Granon, S., Muir, J.L., Durantou, F., Harrison, A., & Everitt, B.J. (1998). Neural systems underlying arousal and attention. Implications for drug abuse. Annals of the New York Academy of Sciences, 846, 222-237.
57.
Rolls, E.T. (2000). The brain and emotion. New York: Oxford University Press.
58.
Salichs, M.A., & Malfaz, M. (2006). Using emotions on autonomous agents. The role of happiness, sadness and fear. In Proceedings of the AISB'06 Symposium on Integrative Approaches to Machine Consciousness (pp. 157-164). Bristol, UK: AISB Press.
59.
Sarter, M., Gehring, W.J., & Kozak, R. (2006). More attention must be paid: the neurobiology of attentional effort. Brain Research Reviews, 51, 145-160.
60.
Sarter, M., Hasselmo, M.E., Bruno, J.P., & Givens, B. (2005). Unraveling the attentional functions of cortical cholinergic inputs: interactions between signal-driven and cognitive modulation of signal detection. Brain Research Reviews, 48, 98-111.
61.
Scatton, B., Simon, H., Le Moal, M., & Bischoff, S. (1980). Origin of dopaminergic innervation of the rat hippocampal formation. Neuroscience Letters, 18, 125-131.
62.
Schall, J.D. (2001). Neural basis of deciding, choosing and acting . Nature Reviews Neuroscience, 2, 33-42.
63.
Schultz, W. (1997). Dopamine neurons and their role in reward mechanisms . Current Opinion in Neurobiology, 7, 191-197.
64.
Schultz, W., Dayan, P., & Montague, P.R. (1997). A neural substrate of prediction and reward. Science, 275, 1593-1599.
65.
Schweighofer, N., Bertin, M., Shishida, K., Okamoto, Y., Tanaka, S.C., Yamawaki, S., & Doya, K. (2008). Low-serotonin levels increase delayed reward discounting in humans. Journal of Neuroscience, 28, 4528-4532.
66.
Schweighofer, N., Tanaka, S.C., & Doya, K. (2007). Serotonin and the evaluation of future rewards: theory, experiments, and possible neural mechanisms. Annals of the New York Academy of Sciences, 1104, 289-300.
67.
Servan-Schreiber, D., Printz, H., & Cohen, J.D. (1990). A network model of catecholamine effects: gain, signal-to-noise ratio, and behavior. Science, 249, 892-895.
68.
Sesack, S.R., & Pickel, V.M. (1992). Prefrontal cortical efferents in the rat synapse on unlabeled neuronal targets of catecholamine terminals in the nucleus accumbens septi and on dopamine neurons in the ventral tegmental area. Journal of Comparative Neurology, 320, 145-160.
69.
Seth, A.K., McKinstry, J.L., Edelman, G.M., & Krichmar, J.L. (2004). Visual binding through reentrant connectivity and dynamic synchronization in a brain-based device . Cerebral Cortex, 14, 1185-1199.
70.
Smith, P.L., & Ratcliff, R. (2004). Psychology and neurobiology of simple decisions . Trends in Neurosciences, 27, 161-168.
71.
Sporns, O., & Alexander, W.H. (2002). Neuromodulation and plasticity in an autonomous robot. Neural Networks, 15, 761-774.
72.
Squyres, S. (2005). Roving Mars : Spirit, opportunity, and the exploration of the Red Planet. New York: Hyperion .
73.
Stone, P., Sutton, R.S., & Kuhlmann, G. (2005). Reinforcement learning for RoboCup soccer keepaway . Adaptive Behavior, 13, 165-188.
74.
Sutton, R.S., & Barto, A.G. (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press.
75.
Tanaka, S.C., Schweighofer, N., Asahi, S., Shishida, K., Okamoto, Y., Yamawaki, S., & Doya, K. (2007). Serotonin differentially regulates short- and long-term prediction of rewards in the ventral and dorsal striatum. PLoS ONE, 2, e1333.
76.
Thrun, S., Burgarde, W., & Fox, D. (2005). Probabilistic robotics. Cambridge, MA: MIT Press.
77.
Usher, M., & McClelland, J.L. (2001). The time course of perceptual choice: the leaky, competing accumulator model. Psychological Review, 108, 550-592.
78.
Williams, G.V., & Goldman-Rakic, P.S. (1995). Modulation of memory fields by dopamine D1 receptors in prefrontal cortex. Nature , 376, 572-575.
79.
Williams, G.V., Rao, S.G., & Goldman-Rakic, P.S. (2002). The physiological role of 5-HT2A receptors in working memory. Journal of Neuroscience , 22, 2843-2854.
80.
Yenne, B. (2004). Attack of the drones. St. Paul, MN: Zenith Press.
81.
Yu, A.J., & Dayan, P. (2005). Uncertainty, neuromodulation, and attention. Neuron, 46, 681-692.
