Diverse, complex, and adaptive animal behaviors are achieved by organizing hierarchically structured controllers in motor systems. The levels of control progress from simple spinal reflexes and central pattern generators through to executive cognitive control in the frontal cortex. Various types of hierarchical control structures have been introduced and shown to be effective in past artificial agent models, but few studies have shown how such structures can self-organize. This study describes how such hierarchical control may evolve in a simple recurrent neural network model implemented in a mobile robot. Topological constraints on information flow are found to improve system performance by decreasing interference between different parts of the network. One part becomes responsible for generating lower behavior primitives while another part evolves top-down sequencing of the primitives for achieving global goals. Fast and slow neuronal response dynamics are automatically generated in specific neurons of the lower and the higher levels, respectively. A hierarchical neural network is shown to outperform a comparable single-level network in controlling a mobile robot.
Aharonov-Barki, R. , Beker, T. , & Ruppin, E. (2001). Emergence of memory-driven command neurons in evolved artificial agents . Neural Computation, 13, 691-716 .
2.
Albert, R. , & Barabasi, A. (2002). Statistical mechanism of complex networks . Reviews of Modern Physics, 74(1), 47-97 .
3.
Arbib, M. A. (1981). Perceptual structures and distributed motor control. In Brooks, V. B. (Ed.), Handbook of physiology (Vol. II, pp. 1449-1480). Bethesda, MD: American Physiological Society .
4.
Blynel, J. (2003). Evolving reinforcement learning-like abilities for robots . In A. Tyrrell , P. C. Haddow , & J. Torresen (Eds.), Evolvable systems: From biology to hardware: Fifth International Conference, ICES 2003,Lecture Notes in Computer Science (pp. 320-331 ), Vol. 2606 Berlin: Springer Verlag.
5.
Blynel, J. , & Floreano, D. (2002). Levels of dynamics and adaptive behavior in evolutionary neural controllers . In B. Hallam , D. Floreano , J. Hallam , G. Hayes , & J. A. Meyer (Eds.), From animals to animats 7: Proceedings of the Seventh International Conference on Simulation of Adaptive Behavior (pp. 272-281 ). Cambridge, MA: MIT Press, Bradford Books.
6.
Crutcher, M. D. , & Alexander, G. E. (1990). Movement-related neuronal activity selectively coding either direction or muscle pattern in 3 motor areas of the monkey . Journal of Neurophysiology, 64(1), 151-163 .
7.
Cyberbotics, Ltd.http://www.cyberbotics.com, Webots Commercial Mobile Robot Simulation Software.
8.
Dietterich, T. (2000). Hierarchical reinforcement learning with the MAXQ value function decomposition . Journal of Artificial Intelligence Research, 13, 227-303 .
9.
Doya K. , Samejima K. , Katagiri K. , & Kawato M. (2002). Multiple model-based reinforcement learning . Neural Computation, 14, 1347-1369 .
10.
Eck, D. , & Schmidhuber, J. (2002). Learning the long-term structure of the blues . In J. R. Dorronsoro (Ed.), Proceedings of the International Conference on Artificial Neural Networks, Lecture Notes in Computer Science (Vol. 2415). Berlin: Springer.
11.
Fan, J. , Yao, Q. , & Tong, H. (1996). Estimation of densities and sensitivity measures in nonlinear dynamical systems . Biometrika, 83(1), 189-206 .
12.
Fikes, R. E. , Hart, P. E. , & Nilsson, N. J. (1972). Learning and executing generalized robot plans . Artificial Intelligence, 3, 251-288 .
13.
Floreano, D. , & Mondada, F. (1994). Automatic creation of an autonomous agent: Genetic evolution of a neural-network driven robot . In D. Cliff , P. Husbands , J. Meyer , & S. W. Wilson (Eds.), From animals to animats 3: Proceedings of the Third Conference on Simulation of Adaptive Behavior. Cambridge, MA: The MIT Press, Bradford Books.
14.
French, R. M. (1991). Using semi-distributed representation to overcome catastrophic forgetting in connectionist networks . In K. J. Hammond & K. J. Genther (Eds.) Proceedings of the 13th Annual Cognitive Science Society Conference (pp. 173-178 ). Hillsdale, NJ: Lawrence Erlbaum Publishers.
15.
Fujimoto, K. , & Kaneko, K. (2003). How fast elements can affect slow dynamics . Physica D: Nonlinear Phenomena, 180, 1-2 .
16.
Gers, F. A. , Schraudolph, N. N. , & Schmidhuber, J. (2002). Learning precise timing with LSTM recurrent networks . Journal of Machine Learning Research, 3, 115-143 .
17.
Ghez, C. , & Krakauer, J. (2000). The organization of movement. In E. R. Kandel , J. H. Schwartz , T. M. Jessell (Eds.), Principles of neural science, (4th ed.), (pp. 653-673). New York: McGraw-Hill .
18.
Giszter, S. F. , Mussa-Ivaldi, F. A. , & Bizzi, E. (1993). Convergent force fields organized in the frog’s spinal cord . Journal of Neuroscience, 13(2), 467-491 .
19.
Goldberg, D. E. (2002). The design of innovation: Lessons from and for competent genetic algorithms. Boston, MA: Kluwer .
20.
Grossberg, S. (1982). Studies of mind and brain: Neural principles of learning, perception, development, cognition, and motor control. Boston Studies in the Philosophy of Science, Vol. 70. Dordrecht: Reidel .
21.
Holland, J. (1975). Adaptation in natural and artificial systems. Ann Arbor, MI: University of Michigan Press .
22.
Ito, M. , & Tani, J. (2004). On-line imitative interaction with a humanoid robot using a dynamic neural network model of a mirror system . Adaptive Behavior, 12(2), 93-115 .
23.
Kelly, J. P. (1991). The neural basis of perception and movement. In E. R. Kandel , J. H. Schwartz , & T. M. Jessell (Eds.), Principles of neural science, (3rd ed.) (pp. 283-295). Norwalk, CN: Appleton and Lange .
24.
Lipson, H. , & Pollack, J. B. (2000). Automatic design and manufacture of robotic lifeforms . Nature, 406, 974-978 .
25.
Lu, X. , & Ashe, J. (2005). Anticipatory activity in primary motor cortex codes memorized movement sequences . Neuron, 45, 967-973 .
26.
McCloskey, M. , & Cohen, N. J. (1989). Catastrophic interference in connectionist networks: The sequential learning problem . The Psychology of Learning and Motivation, 24, 109-165 .
27.
McGovern, A. , & Barto, A. G. (2001). Accelerating reinforcement learning through the discovery of useful subgoals . In Proceedings of the Sixth International Symposium on Artificial Intelligence, Robotics, and Automation in Space: i-SAIRAS, Montreal, Canada, June 2001, Electronically Published.
28.
McGovern, A. , Sutton, R. S. , & Fagg, A. H. (1997). Roles of macro-actions in accelerating reinforcement learning . Proceedings of the 1997 Grace Hopper Celebration of Women in Computing (pp. 13-17 ). San Jose, CA: Anita Borg Institute for Women and Technology.
29.
Miller, G. F. , Todd, P. M. , & Hegde, S. U. (1989). Designing neural networks using genetic algorithms . In J. D. Schaffer (Ed.), Proceedings of the Third International Conference on Genetic Algorithms (pp. 379-384 ). San Mateo, CA: Morgan Kaufmann.
30.
Mitchell, M. (1998). An introduction to genetic algorithms. Cambridge, MA: The MIT Press .
31.
Mozer, M. C. (1992). Induction of multiscale temporal structure. In J. E. Moody , S. J. Hanson , & R. P. Lippmann (Eds.), Advances in neural information processing systems IV (pp. 275-282). San Mateo, CA: Morgan Kaufmann .
32.
Ninokura, Y. , Mushiake, H. , & Tanji, J. (2003). Integration of temporal order and object information in the monkey lateral prefrontal cortex . Journal of Neurophysiology, 10, 1152 .
33.
Nishimoto, R. , & Tani, J. (2004). Learning to generate combinatorial action sequences utilizing the initial sensitivity of deterministic dynamical systems . Neural Networks, 17(7), 925-933 .
34.
Nolfi, S. (2002). Evolving robots able to self-localize in the environment: The importance of viewing cognition as the result of processes occurring at different time scales . Connection Science, 14(3), 231-244 .
35.
Ogata, T. , Sugano, S. , & Tani, J. (2004). Open-end human robot interaction from the dynamical systems perspective: Mutual adaptation and incremental learning . Lecture Notes in Artificial Intelligence (Vol. 3029, pp. 435-444 ), Berlin: Springer.
36.
Paine, R. W. , & Tani, J. (2004). Motor primitive and sequence self-organization in a hierarchical recurrent neural network . Neural Networks, 17, 1291-1309 .
37.
Parr, R. , & Russell, S. (1998). Reinforcement learning with hierarchies of machines. In M. I. Jordan , M. J. Kearns , & S. A. Solla (Eds.), Advances in neural information processing systems (Vol. 10, pp. 1043-1049). Cambridge, MA: The MIT Press .
38.
Precup, D. , & Sutton, R. S. (1997). Multi-time models for temporally abstract planning. In M. I. Jordan , M. J. Kearns , & S. A. Solla (Eds.), Advances in neural information processing systems (Vol. 10, pp. 1050-1056). Cambridge, MA: The MIT Press .
39.
Ruppin, E. (2002). Evolutionary autonomous agents: A neuroscience perspective . Nature Reviews Neuroscience, 3, 132-141 .
40.
Sakai, K. , Kitaguchi, K. , & Hikosaka, O. (2003). Chunking during human visuomotor sequence learning . Experimental Brain Research, 152(2), 229-242 .
41.
Siegelmann, H. T. , & Sontag, E. D. (1995). On the computational power of neural nets . Journal of Computer and System Sciences, 50(1), 132-150 .
42.
Stanley, K. O. , & Mikkulainen, R. (2004). Competitive coevolution through evolutionary complexification . Journal of Artificial Intelligence Research, 21, 63-100 .
43.
Sugita, Y. , & Tani, J. (2005). Learning semantic combinatoriality from the interaction between linguistic and behavioral processes . Adaptive Behavior, 13, 33-52 .
44.
Sutton, R. , Precup, D. , & Singh, S. (1999). Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning . Artificial Intelligence, 112, 181-211 .
45.
Tani, J. (2003). Learning to generate articulated behavior through the bottom-up and the top-down interaction processes . Neural Networks, 16(1), 11-23 .
46.
Tani, J. , & Nolfi, S. (1999). Learning to perceive the world as articulated: An approach for hierarchical learning in sensory-motor systems . Neural Networks12, 1131-1141 .
47.
Tanji, J. , & Kurata, K. (1982). Comparison of movement-related activity in 2 cortical motor areas of primates . Journal of Neurophysiology, 48(3), 633-653 .
48.
Tanji, J. , & Shima, K. (1994). Role for supplementary motor area cells in planning several movements ahead . Nature, 371, 413-416 .
49.
Thoroughman, K. A. , & Shadmehr, R. (2000). Learning of action through combination of motor primitives . Nature, 407, 742-747 .
50.
Todd, P. M. (1991). A connectionist approach to algorithmic composition. In P. M. Todd & D. G. Loy (Eds.), Music and connectionism (pp. 173-193). Cambridge, MA: MIT Press .
51.
Torres, J. J. , Munoz, M. A. , Marro, J , & Garrido, P. L. (2004). Influence of topology on the performance of a neural network . Neurocomputing, 58-60, 229-234 .
52.
Tucci, E. , Quinn, M. , & Harvey, I. (2002). An evolutionary ecological approach to the study of learning behavior using a robot-based model . Adaptive Behavior, 10(3/4), 201-222 .
53.
Wolpert, D. M. , & Kawato, M. (1998). Multiple paired forward and inverse models for motor control . Neural Networks, 11, 1317-1329 .
54.
Yamauchi, B. , & Beer, R. D. (1994). Sequential behavior and learning in evolved dynamical neural networks . Adaptive Behavior, 2(3), 219-246 .
55.
Ziemke, T. , & Thieme, M. (2002). Neuromodulation of reactive sensorimotor mappings as a short-term memory mechanism in delayed response tasks . Adaptive Behavior, 10(3/4), 185-199 .
