Since 1995, numerous Actor–Critic architectures for reinforcement learning have been proposed as models of dopamine-like reinforcement learning mechanisms in the rat’s basal ganglia. However, these models were usually tested in different tasks, and it is then difficult to compare their efficiency for an autonomous animat. We present here the comparison of four architectures in an animat as it per forms the same reward-seeking task. This will illustrate the consequences of different hypotheses about the management of different Actor sub-modules and Critic units, and their more or less autono mously determined coordination. We show that the classical method of coordination of modules by mixture of experts, depending on each module’s performance, did not allow solving our task. Then we address the question of which principle should be applied efficiently to combine these units. Improve ments for Critic modeling and accuracy of Actor–Critic models for a natural task are finally discussed in the perspective of our Psikharpax project—an artificial rat having to survive autonomously in unpre dictable environments.
Aizman, O. , Brismar, H. , Uhlen, P. , Zettergren, E. , Levey, A. I. , Forssberg, H. , Greengard, P. , & Aperia, A. (2000). Anatomical and physiological evidence for D1 and D2 dopamine receptors colocalization in neostriatal neurons . Nature Neuroscience, 3(3), 226–230 .
2.
Albertin, S. V. , Mulder, A. B. , Tabuchi, E. , Zugaro, M. B. , & Wiener, S. I. (2000). Lesions of the medial shell of the nucleus accumbens impair rats in finding larger rewards, but spare reward-seeking behavior . Behavioral Brain Research, 117(1–2), 173–183 .
3.
Albin, R. L. , Young, A. B. , & Penney, J. B. (1989). The functional anatomy of basal ganglia disorders . Trends in Neuroscience, 12, 366–375 .
4.
Arleo, A. , & Gerstner, W. (2000). Spatial cognition and neuromimetic navigation: A model of the rat hippocampal place cell activity . Biological Cybernetics, Special Issue on Navigation in Biological and Artificial Systems, 83, 287–299 .
5.
Baldassarre, G. (2002). A modular neural-network model of the basal ganglia’s role in learning and selecting motor behaviors . Journal of Cognitive Systems Research, 3(1), 5–13 .
6.
Baldassarre, G. , & Parisi, D. (2000). Classical and instrumental conditioning: From laboratory phenomena to integrated mechanisms for adaptation. In J.-A. Meyer , A. Berthoz , D. Floreana , H. L. Roitblat , and S. W. Wilson (Eds.), From animals to animats 6: Proceedings of the Sixth International Conference on Simulation of Adaptive Behavior, supplement volume (pp. 131–139). Cambridge, MA: The MIT Press .
7.
Brown, J. , Bullock, D. , & Grossberg, S. (1999). How the basal ganglia use parallel excitatory and inhibitory learning, or incentive salience?Brain Research Reviews, 28, 309–369 .
8.
Brown, L. , & Sharp. F. (1995). Metabolic mapping of rat striatum: Somatotopic organization of sensorimotor activity . Brain Research, 686, 207–222 .
9.
Bunney, B. S. , Chiodo, L. A. , & Grace, A. A. (1991). Midbrain dopamine system electrophysiological functioning: A review and new hypothesis . Synapse, 9, 79–84 .
10.
Burgess, N. , Jeffery, K. J. , & O’Keefe, J. (1999). Integrating hippocampal and parietal functions: A spatial point of view. In N. Burgess , K. J. Jeffery , and J. O’Keefe (Eds.), The hippocampal and parietal foundations of spatial cognition (pp. 3–29). Oxford: Oxford University Press .
11.
Cardinal, R. N. , Parkinson, J. A. , Hall, J. , & Everitt, B. J. (2002). Emotion and motivation: The role of the amygdala, ventral striatum and prefrontal cortex . Neuroscience Biobehavioral Reviews, 26(3), 321–352 .
12.
Dayan., P. (2001). Motivated reinforcement learning. In T. G.Dietterich, S.Becker , & Z. Ghahramani (Eds.), Proceedings of NIPS 14 (pp. 11–18). Cambridge, MA: The MIT Press .
13.
Daw, N. D. (2003). Reinforcement learning models of the dopamine system and their behavioral implications. Ph.D. thesis, Carnegie Mellon University, Pittsburgh, PA.
14.
Doya, K. (2000). Reinforcement learning in continuous time and space . Neural Computation, 12, 219–245 .
15.
Doya, K. , Samejima, K. , Katagiri, K. , & Kawato, M. (2002) Multiple model-based reinforcement learning . Neural Computation, 14(6), 1347–1369 .
16.
Filliat, D. , Girard, B. , Guillot, A. , Khamassi, M. , Lachèze, L. , & Meyer, J.-A. (2004). State of the artificial rat Psikharpax. In S. Schaal , A. Ijspeert , A. Billard , S. Vijayakumar , J. Hallam , and J.-A. Meyer (Eds.), From animals to animats 8: Proceedings of the Eighth International Conference on Simulation of Adaptive Behavior (pp. 2–12). Cambridge, MA: The MIT Press .
17.
Foster, D. , Morris, R. , & Dayan, P. (2000). Models of hippocampally dependent navigation using the temporal difference learning rule . Hippocampus, 10, 1–16 .
18.
Frank, M. J. , Loughry, B. , & O’Reilly, R. C. (2001). Interactions between frontal cortex and basal ganglia in working memory: A computational model . Cognitive, Affective and Behavioral Neuroscience, 1(2), 137–160 .
19.
Gerfen, C. R. , Herkenham, M. , & Thibault, J. (1987). The neostriatal mosaic: II. Patch- and matrix-directed mesostriatal dopaminergic and non-dopaminergic systems . Journal of Neuroscience, 7, 3915–3934 .
20.
Girard, B. , Cuzin, V. , Guillot, A. , Gurney, K. , & Prescott, T. (2003). A basal ganglia inspired model of action selection evaluated in a robotic survival task . Journal of Integrative Neuroscience, 2(22), 179–200 .
21.
Girard, B. , Filliat, D. , Meyer, J.-A. , Berthoz, A. , & Guillot, A. (2005). Integration of navigation and action selection functionalities in a computational model of cortico-basal-thalamo-cortical loops . Adaptive Behavior, 13(2), 115–130 .
22.
Gurney, K. N. , Prescott, T. J. , & Redgrave, P. (2001a). A computational model of action selection in the basal ganglia: I. A new functional anatomy . Biological Cybernetics, 84, 401–410 .
23.
Gurney, K. N. , Prescott, T. J. , & Redgrave, P. (2001b). A computational model of action selection in the basal ganglia: II. Analysis and simulation of behavior . Biological Cybernetics, 84, 411–423 .
24.
Houk, J. C. , Adams, J. L. , & Barto, A. G. (1995). A model of how the basal ganglia generate and use neural signals that predict reinforcement. In J. C. Houk , J. L. Davis , and D. G. Beiser (Eds.), Models of information processing in the basal ganglia, Cambridge, MA: The MIT Press .
25.
Ikemoto, S. , & Panksepp., J. (1999). The role of the nucleus accumbens dopamine in motivated behavior: A unifying interpretation with special reference to reward-seeking . Brain Research Reviews, 31, 6–41 .
26.
Jacobs, R. A. , Jordan, M. I. , Nowlan, S. J. , & Hinton, G. E. (1991). Adaptive Mixture of Local Experts . Neural Computation, 3, 79–87 .
27.
Joel, D. , Niv, Y. , & Ruppin, E. (2002). Actor–critic models of the basal ganglia: New anatomical and computational perspectives . Neural Networks, 15, 535–547 .
28.
Joel, D. , & Weiner, I. (2000). The connections of the dopaminergic system with striatum in rats and primates: An analysis with respect to the functional and compartmental organization of the striatum . Neuroscience, 96, 451–474 .
29.
Killcross, A. S. , & Coutureau, E. (2003). Coordination of actions and habits in the medial prefrontal cortex of rats . Cerebral Cortex, 13(4), 400–408 .
30.
Lee, J. K. , & Kim, I. H. (2003). Reinforcement learning control using self-organizing map and multi-layer feed-for-ward neural network . In Proceedings of the International Conference on Control Automation and Systems, ICCAS 2003 (pp. 142–145 ). Gyeongju, South Korea.
31.
McNaughton, B. L. (1989). Neural mechanisms for spatial computation and information storage. In L. Nadel , L. A. Cooper , P. Harnish , and R. M. Colicover (Eds.), Neural Connections, Mental Computations (chapter 9, pp. 285–350). Cambridge, MA: MIT Press .
32.
Montague, P. R. , Dayan, P. , & Sejnowski, T. J. (1996). A framework for mesencephalic dopamine systems based on predictive Hebbian learning . Journal of Neuroscience, 16, 1936–1947 .
33.
Montes-Gonzalez, F.Prescott, T. J. , Gurney, K. N. , Humphries, M. , & Redgrave, P. (2000). An embodied model of action selection mechanisms in the vertebrate brain. In J.-A. Meyer , A. Bethoz , D. Floreana , H. L. Roitblat , and S. W. Wilson (Eds.), From animals to animats 6: Proceedings of the Sixth International Conference on Simulation of Adaptive Behavior (pp.157–166). Cambridge, MA: The MIT Press .
34.
O’Doherty, J. , Dayan, P. , Schultz, J. , Deichmann, R. , Friston, K. , & Dolan, R. (2004). Dissociable roles of dorsal and ventral striatum in instrumental conditioning . Science, 304, 452–454 .
35.
Pennartz, C. M. A. (1996). The ascending neuromodulatory systems in learning by reinforcement: Comparing computational conjectures with experimental findings . Brain Research Reviews, 21, 219–245 .
36.
Perez-Uribe, A. (2001). Using a time-delay actor–critic neural architecture with dopamine-like reinforcement signal for learning in autonomous robots. In S. WermterJ. Austin , and D. Willshaw (Eds.), Emergent neural computational architectures based on neuroscience: A state-of-the-art survey (pp. 522–533). Berlin: Springer .
37.
Schultz, W. (1998). Predictive reward signal of dopamine neurons . Journal of Neurophysiology, 80(1), 1–27 .
38.
Schultz, W. , Apicella, P. , & Ljungberg, T. (1993). Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task . Journal of Neuroscience, 13(3), 900–913 .
39.
Schultz, W. , Dayan, P. , & Montague, P. R. (1997). A neural substrate of prediction and reward . Science, 275, 1593–1599 .
40.
Smith, A. J. (2002). Applications of the self-organizing map to reinforcement learning . Neural Networks, 15(8-9), 1107–1124 .
41.
Sporns, O. , & Alexander, W. H. (2002). Neuromodulation and plasticity in an autonomous robot . Neural Networks, 15, 761–774 .
42.
Strösslin, T. (2004). A connectionist model of spatial learning in the rat. Ph.D thesis, EPFL, Swiss Federal Institute of Technology.
43.
Suri, R. E. , & Schultz, W. (2001). Temporal difference model reproduces anticipatory neural activity . Neural Computation, 13, 841–862 .
44.
Sutton, R. S. , & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge, MA: The MIT Press .
45.
Tang, B. , Heywood, M. I. , & Shepherd, M. (2002). Input partitioning to mixture of experts . In IEEE/INNS International Joint Conference on Neural Networks (pp. 227–232 ), Honolulu, Hawaii (pp. 227–232 ).
46.
Tani, J. , & Nolfi, S. (1999). Learning to perceive the world as articulated: An approach for hierarchical learning in sensory-motor systems . Neural Networks, 12(7-8), 1131–1141 .
47.
Thierry, A.-M. , Gioanni, Y. , Dégénetais, E. , & Glowinski, J. (2000). Hippocampo-prefrontal cortex pathway: Anatomical and electrophysiological characteristics . Hippocampus, 10, 411–419 .
48.
Yin, H. H. , Knowlton, B. J. , & Balleine, B. W. (2004). Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning . European Journal of Neuroscience, 19(1), 181–189 .
