Temporal and sequential information is essential to any agent continually interacting with its environment. In this paper, we test whether it is possible to evolve a recurrent neural network controller to match the dynamic requirement of the task. As a benchmark, we consider a sequential navigation task where the agent has to alternately visit two rewarding sites to obtain food and water after first visiting the nest. To achieve a better fitness, the agent must select relevant sensory inputs and update its working memory to realize a non-Markovian sequential behavior in which the preceding state alone does not determine the next action. We compare the performance of a feed-forward and recurrent neural control architectures in different environment settings and analyze the neural mechanisms and environment features exploited by the agents to achieve their goal. Simulation and experimental results using the Cyber Rodent robot show that a modular architecture with a locally excitatory recur rent layer outperformed the general recurrent controller.
Bengio, Y. , Simard, P. , & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult . IEEE Transactions on Neural Networks, 5(2), 157-166 .
2.
Blynel, J. (2003). Evolving reinforcement learning-like abilities for robots . In A. Tyrrell , P. C. Haddow , & J. Torresen (Eds.), Proceedings of 5th International Conference on Evolvable Systems: From Biology to Hardware (pp. 320-331 ). Springer-Verlag GmbH.
3.
Blumberg, B. (1994). Action-selection in hamsterdam: Lessons from ethology . In D. Cliff , P. Husbands , J.-A. Meyer , & W. S (Eds.), From animals to animats 3: The Third International Conference on Simulation of Adaptive Behavior (pp. 108-117 ). Cambridge, MA: MIT Press.
4.
Capi, G. , Uchibe, E. , & Doya, K. (2002). Selection of neural architecture and the environment complexity. In K. Murase & T. Asakura (Eds.), International series on advanced intelligence: Vol. 6, Dynamic systems approach for embodiment and sociality (pp. 311-317).
5.
Doya, K. (1995). Recurrent networks: supervised learning. In M. A. Arbib (Ed), The Handbook of Brain Theory and Neural Networks (pp. 796-799). Cambridge, MA: MIT Press .
6.
Doya, K. (2002). Recurrent networks: learning algorithms. In M. A. Arbib (Ed), The Handbook of Brain Theory and Neural NetworksCambridge, MA: MIT Press .
7.
Doya, K. & Yoshizawa, S. (1989). Adaptive neural oscillator using continuous-time back propagation learning . Neural Networks, 2, 375-386 .
8.
Elman, J. (1990). Finding structure in time . Cognitive Science, 14, 179-211 .
9.
Elwing, S. , Uchibe, E. , & Doya, K. (2003). An evolutionary approach to automatic construction of the structure in hierarchical reinforcement learning . Proceedings of the Genetic and Evolutionary Computation Conference, ( 507-509 ). Springer-Verlag Berlin Heidelberg.
10.
Faihe, Y. , & Muller, J. (1997). Analysis and design of robot’s behavior: towards a methodology . In Proceedings of Sixth European Workshop on Learning Robot, Brighton, UK (pp. 46-61 ). London: Springer-Verlag.
11.
Fujita, I. , Tanaka, K. , Ito, M. , & Cheng, K. (1992). Columns for visual features of objects in monkey inferotemporal cortex . Nature, 360, 343-346 .
12.
Hochreiter, S. , (1997). Recurrent neural net learning and vanishing gradient . In Proceedings of Fuzzy-Neuro Workshop, Soest, Germany.
13.
Jones, J. , & Mataric, M. (1998). Sequential learning in mini-malist neural network . In Proceedings of International Conference on Robotics and Automation (pp. 395-396 ).
14.
Kritzer M. F. , & Goldman-Rakic P. S. (1995). Intrinsic circuit organization of the major layers and sublayers of the dorsolateral prefrontal cortex in the rhesus monkey . Comparative Neurology, 359(1), 131-143 .
15.
Mathayomchan, B. , & Beer, R. D. (2002). Center-crossing recurrent neural networks for the evolution of rhythmic behavior . Neural Computation, 14, 2043-2051 .
16.
Meeden, L. A. (1996). An incremental approach to developing intelligent neural controllers for robots . IEEE Transaction on Systems, Man, and Cybernetics, 26, 474-485 .
17.
Michalewich, Z. (1994). Genetic algorithms + data structures = evaluation programs. Berlin: Springer .
18.
Miyazaki, K. , Miyazaki, K. , Mogi, E. , Matsumoto, G. , & Tetsuya, K. (2002). Role of the nucleus accumbens neurons in goal-directed behavior . In Proceedings of the International Symposium on Limbic and Association Cortical Systems—Basic, Clinical and Computational Aspects, Toyama, Japan.
19.
Nakahara, H. , Doya, K. , & Hikosaka, O. (2001). Parallel cortico-basal ganglia mechanisms for acquisition and execution of visuomotor sequences—A computational approach . Journal of Cognitive Neuroscience, 13, 626-647 .
20.
Nakamura, K. , Sakai, K. , & Hikosaka O. (1998). Neuronal activity in medial frontal cortex during learning of sequential procedures . Journal of Neurophysiology, 80(5), 2671-2687 .
21.
Paul, C. (2003). Bilateral decoupling in the neural control of biped locomotion . In Proceedings of the Second International Symposium on Adaptive Motion of Animals and Machines, Kyoto, Japan.
22.
Pearlmutter, B. A. (1989). Learning state space trajectories in recurrent neural networks . Neural Computation, 1, 263-269 .
23.
Redgrave, P. , Prescott, T. J. , & Gurney, K. (1999). The basal ganglia: A vertebrate solution to the selection problem?Neuroscience, 89, 1009-1023 .
24.
Rodriguez, P. (2001). Simple recurrent networks learn context-free and context-sensitive languages by counting . Neural Computation, 13, 2093-2118 .
25.
Sato, M. , Nakamura, Y. , & Ishii, S. (2002). Reinforcement learning for biped locomotion . In Proceedings of International Conference on Artificial Neural Networks (pp. 777-782 ).
26.
Seth, A. K. (1998) Evolving action selection and selective attention without actions, attentions, or selection . In R. Pfeifer , B. Blumberg , J. Meyer , & S. Wilson (Eds.), From animals to animats 5: Proceedings of the Fifth International Conference on the Simulation of Adaptive Behavior (pp. 139-147 ). Cambridge, MA: MIT Press.
27.
Tuci, E. , Quinn, M. , & Harvey, I. (2002). An evolutionary ecological approach to the study of learning behavior using a robot-based model . Adaptive Behavior, 10, 201-221 .
28.
Urzelai, J. , & Floreano, D. (2001). Evolution of adaptive synapses: robots with fast adaptive behavior in new environments . Evolutionary Computation, 9, 495-524 .
29.
Williams R. J. , & Zipser, D. (1989). A learning algorithm for continually running fully recurrent neural networks . Neural Computation, 1, 270-280 .
30.
Yamauchi, B. , & Beer, R. (1996). Sequential behavior and learning in evolved dynamical neural networks . Adaptive Behavior, 2(3), 219-246 .
