The learning architecture described in this article autonomously acquires a topographical (metric) map that encodes a measure of "value" for xy-Cartesian locations in an environment. There are two reasons for the creation of low value areas. Direct negative reinforcement from the environment will result from the robot discovering obstacles or having other "unpleasant" experiences.- The other source of negative reinforcement is internally generated by the learning algorithm, as it identifies regions that are a long distance away from the "pleasant" places in the environment. Conversely example "pleasant" places, where positive environmental reward is received, might be energy-charging sites or simply locations that the robot should visit in executing its daily tasks. In general what the robot learns is a map of "motivational" tendencies, or "expectancies". In such a map, the value attached to a place comes to reflect a balance between the good and bad rewards attainable from that position. When the Temporal Difference learning part of the architecture is turned on, that measure of value comes to include an estimate of how far, in travel time, it is to positive reinforcement. The architecture is loosely based on an Adaptive Heuristic Critic structure. Exploration of a continuous-valued search space is conducted by an Evolution Strategy, tuned for fast and approximate optimization. Knowledge acquired autonomously from this exploration is stored in a Radial Basis Function (RBF) neural network. Inherent features of this neural network type lead to the creation of a "potential field" structure that exerts appetitive and aversive "forces"on the robot as it moves around in the environment. The results of simulation experiments are presented, with a view to illustrating the strengths and weaknesses of the architecture. The map building architecture proposed here is intended to form part of an overall navigational system. In future work it will be integrated with a self-localization algorithm, landmark-based topological mapping, and a reactive system for dealing with local dynamics in the environment.
Arbib, M.A. (1995). Schema Theory, The Handbook of Brain Theory and Neural Networks, Ed. Arbib M A, MIT Press, pp.830-834.
2.
Arbib, M.A. (1981). Perceptual structures and distributed motor control, Handbook of physiology- the nervous system, II, Ed. Brooks V B, American Physiological Society, pp.1449-1465.
3.
Arkin, R.C. (1992). Behavior-based robot navigation for extended domains , Journal of Adaptive Behavior, 1 (2), MIT Press, pp.201-225.
4.
Arkin, R.C. (1989). Motor schema-based mobile robot navigation, International Journal of Robotics Research, vol. 8, no. 4, MIT Press, pp.92-112.
5.
Arkin, R.C. (1987). Towards cosmopolitan robots: Intelligent navigation in extended man-made environments, PhD thesis, COINS technical report 87-80, University of Massachusetts, Dept. of Computer and Information Science.
6.
Bäck, T. & Schwefel, H-P. (1993). An overview of evolutionary algorithms for parameter optimization, Evolutionary Computation vol. 1, no. 1, pp.1-23.
7.
Beyer, H.-G. (1995). Towards a theory of evolution strategies: On the benefits of sex - the ( /, ) theory, Journal of Evolutionary Computation, 3(1), pp.81-111.
8.
Brooks, R.A. (1991). Intelligence without representation, Journal of Artificial Intelligence, vol.47, pp.139-159.
9.
Broomhead, D.S. & Lowe, D. (1988). Multivariable function interpolation and adaptive network, Complex Systems2, pp.321-355.
10.
Chown, E. (1999). Making predictions in an uncertain world: Environmental structure and cognitive maps, Journal of Adaptive Behavior , 7 (1), International Society for Adaptive Behavior, pp.17-33.
11.
Clark, A. & Grush, R. (1999). Towards a cognitive robotics, Journal of Adaptive Behavior, 7 (1), International Society for Adaptive Behavior , pp.5-16.
12.
Clark, A. & Wheeler, M. (1998). Bringing representation back to life, From Animals to Animats 5, Proceedings of Fifth International Conference on Simulation of Adaptive Behavior, pp.3-12.
13.
Collett, T.S., Cartwright, B.A. & Smith, B.A. (1986). Landmark learning and visuo-spatial memories in gerbils, Journal of Comparative Physiology A158, pp.835-851.
14.
Connell, J.H. (1992). SSS: A hybrid architecture applied to robot navigation , Proceedings of IEEE International Conference on Robotics and Automation, pp.2719-2724.
15.
Donnart, J-Y. & Meyer, J-A. (1997). Hierarchical map building and self-positioning with MonaLysa, Journal of Adaptive Behavior, 5 (1), MIT Press , pp.29-74.
16.
Donnart, J-Y. & Meyer, J-A. (1994). A hierarchical classifier system implementing a motivationally autonomous animat, From Animals to Animats 3, Proceedings of Third International Conference on Simulation of Adaptive Behavior, MIT Press, pp.144-153.
17.
Etienne, A.S. (1992). Navigation of a small mammal by dead reckoning and local cues, Current Directions in Psychological Science , vol. 1, no. 2, pp.48-52.
18.
Gallistel, C.R. (1990). The Organization of Learning, MIT Press.
19.
Grasso, F., Consi, T., Mountain, D. & Atema, J. (1996). Locating odor sources in turbulence with a lobster inspired robot, Proceedings of Fourth International Conference on Simulation of Adaptive Behavior, MIT Press , pp.104-112.
20.
Guazzelli, A., Corbacho, F.J., Bota, M. & Arbib, M.A. (1998). Affordances, motivations, and the world graph theory, Journal of Adaptive Behavior, 6(3-4), MIT Press, pp.435-471.
21.
Hoffmeister & Bäck. (1990). Genetic algorithms and evolutionary strategies: Similarities and differences, Procs. of Conference on Parallel Problem Solving from Nature 1, pp.455-469.
22.
Klopf, A.H., Morgan, J.S. & Weaver, S.E. (1993). A Hierarchical network of control systems that learn: modelling nervous system function during classical and instrumental conditioning, Journal of Adaptive Behavior, 1(3), MIT Press , pp.263-319.
23.
Klopf, A.H. (1988). A neuronal model of classical conditioning, Psychobiology, 16(2), pp.85-125.
24.
Kortenkamp, D., Bonasso, R.P., Murphy R. (1998). Artificial Intelligence and Mobile Robotics, AAAI Press/MIT Press, ISBN 0-262-61137-6.
25.
Kortenkamp, D., Chown, E. & Kaplan, S. (1996). Implementing a psychological model of cognitive mapping on a mobile robot: From theory to practice. Submitted.
26.
Kortenkamp, D., Huber, M., Cohen, C., Raschke, U., Koss, F. & Congdon, C. (1998). In Artificial Intelligence and Mobile Robotics , Eds. Kortenkamp, D., Bonasso, R. P., Murphy R. , AAAI Press/MIT Press, ISBN 0-262-61137-6, pp.53-71.
27.
Kuipers, B. & Byun, Y.T. (1991). A robot exploration and mapping strategy based on a semantic hierarchy of spatial representations, Robotics and Autonomous Systems8.
28.
Li, G., Svensson, B. & Lansner, A. (1998). Self-orienting with on-line learning of environmental features, Journal of Adaptive Behavior, 6 (3-4), MIT Press, pp.535-566 .
29.
Lin, L.-J. (1993). Reinforcement learning for robots using neural networks, PhD thesis, School of Computer ScienceCarnegie Mellon University, Pittsburgh, USA.
30.
MacFarlane, D.A. (1930). The role of kinesthesis in maze learning, University of California Publications in Psychology, 4, pp.277-305.
31.
Mataric, M.J. (1992). Integration of representation into goal-driven behavior-based robots, IEEE Transactions on Robotics and Automation , vol. 8, No. 3, pp.304-312.
32.
McNaughton, B.L. (1988). Neural mechanisms for spatial computation and information storage, Neural Connections and Mental Computations, Eds. Nadel L, Cooper L A, Culicover P & Harnish R M, MIT Press, Cambridge, Mass. , pp.285-350.
33.
Moravec, H.P. & Elfes, A. (1985). High resolution maps from wide angle sonar, Proceedings of IEEE International Conference on Robotics & Automation, pp.116-121.
34.
Muller, R.U., Kubie, J.L., Bostock, E.M., Taube, J.S. & Quirk, G.J. (1990). Spatial firing correlates of neurons in the hippocampus of freely moving rats, Brain and Space, Oxford: Oxford University Press, pp.286-333.
35.
O'Keefe, J. (1990). The hippocampal cognitive map and navigational strategies, Brain and Space, Oxford University Press.
36.
O'Keefe, J. & Nadel, L. (1978). The hippocampus as a cognitive map, Oxford University Press.
37.
Ostermeier, A., Gawelczyk, A. & Hansen, N. (1995). A derandomised approach to self-adaptation of evolution strategies, Journal of Evolutionary Computation , 2(4), pp.369-380, MIT Press.
38.
Pipe, A.G. (1997). Reinforcement Learning and Knowledge Transformation in Mobile Robotics, PhD thesis, Faculty of Engineering, University of theWest of England, Bristol, UK.
39.
Pipe, A.G., Carse, B. & Winfield, A. (1997). A topological map based navigation system for mobile robotics, Proceedings of IEEE International Conference on Systems, Man & Cybernetics, Florida, USA, pp.4394-4399.
40.
Pipe, A.G., Carse, B., Fogarty, T. & Winfield, A. (1995). Learning subjective "fuzzy cognitive maps" in the presence of sensory-motor errors, Proceedings of Third European Conference on Artificial Life, Granada , Spain.
41.
Pipe, A.G., Fogarty, T.C. & Winfield, A. (1994). A Hybrid architecture for learning continuous environmental models in maze problems, From Animals to Animats 3, Proceedings of Third International Conference on Simulation of Adaptive Behavior, MIT Press, pp.198-205.
42.
Poggio, T. & Girosi, F. (1989). A theory of networks for approximation and learning, AI lab. Memo 1140MITCambridge, MA.
43.
Prescott, T.J. (1996). Spatial representation for navigation in animats , Journal of Adaptive Behavior, 4 (2), MIT Press, pp.85-123.
44.
Prescott, T.J. & Mayhew, J.E.W. (1992). Building long-range cognitive maps using local landmarks, From Animals to Animats 2, Proceedings of the Second International Conference on Simulation of Adaptive Behavior, MIT Press, Eds. Meyer J-A, Roitblat H L, Wilson S W, pp.233-242.
45.
Roitblat, H.L. (1990). Cognitive action theory as a control architecture, From Animals to Animats, Proceedings of the First International Conference on Simulation of Adaptive Behavior, MIT Press, Eds. Meyer J.-A, Wilson S W, pp.444-450.
46.
Rudy, J.W., Stadler-Morris, S. & Albert, P. (1987). Ontogeny of spatial navigation behaviors in the rat: dissociation of "proximal"- and "distal"-cuebased behaviors, Behavioral Neuroscience, vol. 101, no. 1, pp.62-73.
47.
Santamaria, J.C., Sutton, R.S. & Ram, A. (1998). Experiments with reinforcement learning in problems with continuous state and action spaces, Journal of Adaptive Behavior, 6 (2), MIT Press, pp. 163-217.
48.
Schmajuk, N.A. & Blair, H.T. (1993). Place learning and the dynamics of spatial navigation: a neural network approach, Journal of Adaptive Behavior, MIT Press, 1 (3), pp.353-385.
49.
Schultz, A.C., Adams, W. & Yamauchi, B. (1999). Integrating exploration, localization, navigation and planning with a common representation, Autonomous Robots6, Kluwer Academic, pp.293-308.
50.
Sullivan, J.C.W. & Pipe, A.G. (1996). An evolution strategy for on-line optimization of dynamic objective functions, Proceedings of Parallel Problem Solving from Nature IV. Lecture Notes in Computer Science #1141, Springer Verlag, Berlin, Germany, pp.781-792.
51.
Sutton, R.S. (1984). Temporal Credit Assignment in Reinforcement Learning , PhD thesis, Dept. of computer and Information Science, University of Massachusetts.
52.
Sutton, R.S. (1988). Learning to predict by the methods of temporal differences, Machine Learning, vol.3, pp.9-44.
53.
Thrun, S., Burgard, W. & Fox, D. (1998). A probabilistic approach to concurrent mapping and localization for mobile robots, Autonomous Robots5, Kluwer Academic, pp.253-271.
54.
Thrun, S., Bücken, A., Burgard, W., Fox, D., Fröhlinghaus, T., Hennig, D., Hofmann, T., Krell, M. & Schmidt, T. (1998). In Artificial Intelligence and Mobile Robotics , Eds. Kortenkamp, D., Bonasso, R. P., Murphy R. , AAAI Press/MIT Press, ISBN 0-262-61137-6, pp.21-52.
55.
Tolman, E.C. (1959). Principles of purposive behavior, Psychology: A Study of Science, Ed. Koch S, New York: McGraw-Hill, pp.92-157.
56.
Tolman, E.C. (1948). Cognitive maps in rats and men, Psychological Review, 55, pp.189-208.
57.
Touretzky, D.S., Wan, H.S. & Redish, A.D. (1994). Neural representation of space in rats and robots, Computational Intelligence: Imitating Life, Eds. Zurada J M, Marks II R J & Robinson C J, IEEE Press, pp.57-68.
58.
Yamauchi, B., Schultz, A. & Adams, W. (1998). Mobile robot exploration and map-building with continuous localization, Proceedings of IEEE International Conference on Robotics & Automation, pp.3715-3720.
59.
Zipser, D. (1985). A computational model of hippocampal place fields , Behavioral Neuroscience, 99, pp.1006-1018.
