This article investigates how a motivational module can drive an animat to learn a sensorimotor cognitive map and use it to generate flexible goal-directed behavior. Inspired by the rat’s hippocampus and neighboring areas, the time growing neural gas (TGNG) algorithm is used, which iteratively builds such a map by means of temporal Hebbian learning. The algorithm is combined with a motivation module, which activates goals, priorities, and consequent activity gradients in the developing cognitive map for the self-motivated control of behavior. The resulting motivated TGNG thus combines a neural cognitive map learning process with top-down, self-motivated, anticipatory behavior control mechanisms. While the algorithms involved are kept rather simple, motivated TGNG displays several emergent behavioral patterns, self-sustainment, and reliable latent learning. We conclude that motivated TGNG constitutes a solid basis for future studies on self-motivated cognitive map learning, on the design of further enhanced systems with additional cognitive modules, and on the realization of highly adaptive, interactive, goal-directed, cognitive systems.
Arbib, M., Bonaiuto, J., Jacobs, S., & Frey, S. ( 2009). Tool use and the distalization of the end-effector . Psychological Research, 73(4), 441-462.
2.
Arleo, A. & Gerstner, W. ( 2000). Spatial cognition and neuro-mimetic navigation: A model of hippocampal place cell activity. Biological Cybernetics , 83, 287-299.
3.
Ay, N., Bertschinger, N., Der, R., Guttler, F., & Olbrich, E. ( 2008). Predictive information and explorative behavior of autonomous robots. The European Physical Journal B - Condensed Matter , 63, 329-339.
4.
Bertsekas, D. ( 1989). Parallel and distributed computation: Numerical methods . Englewood Cliffs, NJ: Prentice-Hall International.
5.
Bishop, C.M., Hinton, G.E., & Strachan, I.G.D. (1997). GTM through time.Proceedings of the IEEE 5th International Conference on Artificial Neural Networks (pp. 111-116).
6.
Bonarini, A., Lazaric, A., & Restelli, M. ( 2006). Incremental skill acquisition for self-motivated learning animats. From Animals to Animats, 9 (pp. 357-368).
Burgess, N., Jackson, A., Hartley, T., & O’Keefe, J. (2000). Predictions derived from modelling the hippocampal role in navigation. Biological Cybernetics , 83, 301-312.
9.
Burgess, N., Recce, M., & O’Keefe, J. (1994). A model of hippocampal function. Neural Networks, 7, 1065-1081.
10.
Butz, M.V. ( 2002). Biasing exploration in an anticipatory learning classifier system. In P. L. Lanzi, W. Stolzmann, & S. W. Wilson (Eds.), Advances in Learning Classifier Systems: 4th International Workshop, IWLCS 2001, LNAI 2321 (pp. 3-22). Berlin Heidelberg: Springer-Verlag.
11.
Butz, M.V. & Reif, K. ( 2010). Motivated TGNG: Algorithm and performance evaluations. Technical Report No. CoboslabY2010N001. Würzburg, Germany: COBOSLAB, Department of Psychology, University of Würzburg . (http://www.coboslab.psychologie.uni-wuerzburg.de )
12.
Butz, M.V., Reif, K., & Herbort, O. ( 2008). Bridging the gap: Learning sensorimotor-linked population codes for planning and motor control. International Conference on Cognitive Systems, CogSys2008.
13.
Butz, M.V., Swarup, S., & Goldberg, D.E. (2004). Effective online detection of task-independent landmarks. IlliGAL Report No. 2004002. University of Illinois at Urbana-Champaign: Illinois Genetic Algorithms Laboratory .
14.
Cañamero, L.D. ( 1997). Modeling motivations and emotions as a basis for intelligent behavior. In W. L. Johnson (Ed.), Proceedings of the 1st International Symposium on Autonomous Agents (Agents’97) (pp. 148-155). New York: The ACM Press.
15.
Capdepuy, P., Polani, D., & Nehaniv, C.L. ( 2007). Construction of an internal predictive model by event anticipation. In M. V. Butz, O. Sigaud, G. Pezzulo, & G. Baldassarre (Eds.), Anticipatory behavior in adaptive learning systems: From brains to individual and social behavior (pp. 218-232). Berlin Heidelberg: Springer-Verlag.
16.
Carpenter, G.A., Grossberg, S., Markuzon, N., Reynolds, J.H., & Rosen, D.B. ( 1992). Fuzzy ARTMAP: A neural network architecture for incremental supervised learning of analog multidimensional maps. IEEE Transactions on Neural Networks, 3(5), 698-713.
17.
Crabbe, F.L. ( 2007). Compromise strategies for action selection. Philosophical Transactions of the Royal Society B: Biological Sciences , 362(1485), 1559-1571.
18.
Cuperlier, N., Quoy, M., Giovannangeli, C., Gaussier, P., & Laroque, P. ( 2006). Transition cells for navigation and planning in an unknown environment. From Animals to Animats, 9 (pp. 286-297).
19.
Der, R. & Martius, G. ( 2006). From motor babbling to purposive actions: Emerging self-exploration in a dynamical systems approach to early robot development . From Animals to Animats, 9 (pp. 406-421).
20.
Duckett, T., Marsland, S., & Shapiro, J. ( 2002). Fast, on-line learning of globally consistent maps. Autonomous Robots, 12(3), 287-300.
21.
Dudek, G., Jenkin, M., Milios, E., & Wilkes, D. ( 1991). Robotic exploration as graph construction. IEEE Transactions on Robotics and Automation, 7, 859-865.
Fellous, J.-M. & Arbib, M.A. ( 2005). Who needs emotions: The brain meets the robot . Oxford, GB: Oxford University Press.
24.
Fleischer, J.G. ( 2007). Neural correlates of anticipation in cerebellum, basal ganglia, and hippocampus. In M. V. Butz , O. Sigaud, G. Pezzulo, & G. Baldassarre (Eds.), Anticipatory behavior in adaptive learning systems: From brains to individual and social behavior (pp. 19-34). Berlin Heidelberg: Springer-Verlag.
25.
Foster, D.J., Morris, R.G.M., & Dayan, P. ( 2000). A model of hippocampally dependent navigation, using the temporal difference learning rule. Hippocampus, 10(1), 1-16.
26.
Foster, D.J. & Wilson, M.A. ( 2006). Reverse replay of behavioural sequences in hippocampal place cells during the awake state. Nature, 440, 680-683.
27.
Franzius, M., Vollgraf, R., & Wiskott, L. ( 2007). From grids to places.Journal of Computational Neuroscience, 22(3), 297-299.
28.
Fritzke, B. ( 1995). A growing neural gas network learns topologies . In G. Tesauro, D. S. Touretzky, & T. K. Leen (Eds.), Advances in neural information processing systems 7 (pp. 625-632). Cambridge MA: MIT Press.
29.
Gadanho, S.C. & Hallam, J. ( 2001). Robot learning driven by emotions. Adaptive Behavior, 9, 42-64.
30.
Gaussier, P., Revel, A., Banquet, J.P., & Babeau, V. ( 2002). From view cells and place cells to cognitive map learning: Processing stages of the hippocampal system. Biological Cybernetics , 86(1), 15-28.
31.
Greenwald, A. ( 1970). Sensory feedback mechanisms in performance control: with special reference to the ideo-motor mechanism. Psychological Review, 77, 73-99.
32.
Herbart, J.F. ( 1825). Psychologie als Wissenschaft neu gegründet auf Erfahrung, Metaphysik und Mathematik. Zweiter, analytischer Teil. Königsberg, Germany: August Wilhem Unzer.
33.
Hoffmann, J., Berner, M., Butz, M.V., Herbort, O., Kiesel, A., Kunde, W. et al. (2007). Explorations of anticipatory behavioral control (ABC): A report from the cognitive psychology unit of the University of Würzburg . Cognitive Processing, 8, 133-142.
34.
Hok, V., Save, E., Lenck-Santini, P.P., & Poucet, B. ( 2005). Coding for spatial goals in the prelimbic/infralimbic area of the rat frontal cortex.Proceedings of the National Academy of Sciences of the USA, 102(12), 4602-4607.
35.
Hommel, B., Müsseler, J., Aschersleben, G., & Prinz, W. ( 2001). The theory of event coding (TEC): A framework for perception and action planning. Behavioral and Brain Sciences , 24, 849-878.
36.
Hsiao, K.Y. & Roy, D. ( 2005). A habit system for an interactive robot. In C. Castelfranchi, C. Balkenius , M. V. Butz, & A. Ortony (Eds.), From reactive to anticipatory cognitive embodied systems: Papers from the AAAI fall symposium (pp. 83-90). Menlo Park, CA: AAAI Press.
37.
Humphries, M.D., Gurney, K., & Prescott, T.J. (2007). Is there a brainstem substrate for action selection?Philosophical Transactions of the Royal Society B, 362, 1627-1639.
38.
Humphrys, M. ( 1996). Action selection methods using reinforcement learning.From Animals to Animats 4: 4th International Conference on Simulation of Adaptive Behavior, SAB-96 (pp. 135-144).
39.
James, W. ( 1890). The principles of psychology. New York: Dover Publications.
40.
Johnson, A. & Redish, A.D. ( 2007). Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. Journal of Neuroscience , 27(45), 12176-12189.
41.
Konczak, J. & Dichgans, J. ( 1997). The development toward stereotypic arm kinematics during reaching in the first 3 years of life. Experimental Brain Research, 117, 346-354.
42.
Konidaris, G.D. & Barto, A. ( 2006). An adaptive robot motivational system. From Animals to Animats, 9 (pp. 346-356).
43.
Konidaris, G.D. & Hayes, G.M. ( 2005). An architecture for behavior-based reinforcement learning . Adaptive Behavior, 13(1), 5-32.
44.
Kortenkamp, D. & Weymouth, T. ( 1994). Topological mapping for mobile robots using a combination of sonar and vision sensing. In AAAI’94: Proceedings of the 12th National Conference on Artificial Intelligence (Vol. 2, pp. 979-984). Menlo Park, CA, USA: American Association for Artificial Intelligence.
Kuipers, B. & Byun, Y. ( 1991). A robot exploration and mapping strategy based on a semantic hierarchy of spatial representations. Robotics and Autonomous Systems, 8, 47-63.
47.
LaValle, S.M. ( 2006). Planning algorithms. Cambridge , UK: Cambridge University Press. (Available at http://planning.cs.uiuc.edu/)
48.
Lisman, J. & Redish, A. ( 2009). Prediction, sequences and the hippocampus. Philosophical Transactions of the Royal Society B: Biological Sciences , 364(1521), 1193-1201.
49.
Maes, P. ( 1991). A bottom-up mechanism for behavior selection in an artificial creature. Proceedings of the 1st International Conference on Simulation of Adaptive Behavior on From Animals to Animats (pp. 238-246).
50.
Maes, P. & Brooks, R.A. ( 1990). Learning to coordinate behaviors. Proceedings of the AAAI-90 (pp. 796-802).
51.
Mataric, M.J. & Brooks, R.A. ( 1990). Learning a distributed map representation based on navigation behaviors. In Proceedings of the USA-Japan Symposium on Flexible Automata (pp. 499-506). New York: ASME .
52.
McNaughton, B.L., Battaglia, F.P., Jensen, O., Moser, E.I., & Moser, M.-B. ( 2006). Path integration and the neural basis of the "cognitive map". Nature Reviews Neuroscience, 7, 663-678.
53.
Merrick, K. & Maher, M.L. ( 2009). Motivated learning from interesting events: Adaptive, multitask learning agents for complex environments. Adaptive Behavior, 17, 7-27.
54.
Möller, R. & Schenck, W. ( 2008). Bootstrapping cognition from behavior-a computerized thought experiment. Cognitive Science, 32(3), 504-542.
55.
Moser, E.I., Kropff, E., & Moser, M.-B. ( 2008). Place cells, grid cells, and the brain’s spatial representation system. Annual Review of Neuroscience , 31(1), 69-89.
56.
O’Keefe, J. & Nadel, L. ( 1978). The hippocampus as a cognitive map. Oxford: Clarendon Press.
57.
Olsson, L.A. ( 2006). Information self-structuring for developmental robotics: Organization, adaptation, and integration. Unpublished doctoral dissertation, School of Computer Science, Faculty of Engineering and Information Sciences , University of Hertfordshire.
58.
Oudeyer, P.-Y., Kaplan, F., & Hafner, V.V. ( 2007). Intrinsic motivation systems for autonomous mental development . IEEE Transactions on Evolutionary Computation , 11, 265-286.
59.
Pierce, D. & Kuipers, B.J. ( 1997). Map learning with uninterpreted sensors and effectors . Artificial Intelligence, 92, 169-227.
60.
Posner, M.I. & Cohen, Y. ( 1984). Components of visual orienting. Attention and Performance X: Control of Language Processes (pp. 551-556).
61.
Poucet, B., Lenck-Santini, P.P., Hok, V., Save, E., Banquet, J.P., Gaussier, P. et al. (2004). Spatial navigation and hippocampal place cell firing: The problem of goal encoding. Reviews in Neurosciences , 15, 89-107.
62.
Prescott, T.J., Redgrave, P., & Gurney, K. ( 1999). Layered control architectures in robots and vertebrates . Adaptive Behavior, 7(1), 99-127.
63.
Remolina, E. & Kuipers, B. ( 2004). Towards a general theory of topological maps. Artificial Intelligence, 152(1), 47- 104.
64.
Renaudineau, S., Poucet, B., & Save, E. ( 2007). Flexible use of proximal objects and distal cues by hippocampal place cells. Hippocampus, 17(5), 381-395.
65.
Rolls, E.T. & Kesner, R.P. ( 2006). A computational theory of hippocampal function, and empirical tests of the theory. Progress in Neurobiology , 79, 1-48.
66.
Saigusa, T., Tero, A., Nakagaki, T., & Kuramoto, Y. ( 2008). Amoebae anticipate periodic events. Physical Review Letters, 100, 018101.
67.
Schill, K., Zetzsche, C., & Hois, J. ( 2009). A belief-based architecture for scene analysis: From sensorimotor features to knowledge and ontology.Fuzzy Sets and Systems, 160(10), 1507-1516.
68.
Schmidhuber, J. ( 1991). A possibility for implementing curiosity and boredom in model-building neural controllers. Proceedings of the 1st International Conference on Simulation of Adaptive Behavior: From Animals to Animats (pp. 222-227).
69.
Seward, J.P. ( 1949). An experimental analysis of latent learning. Journal of Experimental Psychology, 39, 177-186.
70.
Shirinov, E. & Butz, M.V. ( 2009). Distinction between types of motivations: Emergent behavior with a neural, model-based reinforcement learning system. Proceedings of 2009 IEEE Symposium Series on Artificial Life (ALIFE 2009) (pp. 69-76).
71.
Simsek, Ö. & Barto, A.G. ( 2004). Using relative novelty to identify useful temporal abstractions in reinforcement learning. Proceedings of the 21st International Conference on Machine Learning (ICML 2004) (pp. 751-758).
72.
Solstad, T., Boccara, C.N., Kropff, E., Moser, M.-B., & Moser, E.I. ( 2008). Representation of geometric borders in the entorhinal cortex. Science, 322, 1865-1868.
73.
Sprague, N. & Ballard, D.H. ( 2003). Multiple-goal reinforcement learning with modular sarsa(0) . Proceedings of the 18th International Joint Conference on Artificial Intelligence, IJCAI 03 (pp. 1445-1447).
74.
Stringer, S.M., Rolls, E.T., Trappenberg, T.P., & de Araujo, I.E.T. (2002). Self-organizing continuous attractor networks and path integration: Two-dimensional models of place cells . Network: Computation in Neural Systems, 13, 429-446.
75.
Sutton, R.S. ( 1990). Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. Proceedings of the 7th International Conference on Machine Learning (pp. 216-224).
76.
Sutton, R.S. & Barto, A.G. ( 1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press.
77.
Sutton, R.S., Precup, D., & Singh, S. ( 1999). Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence , 112, 181-211.
78.
Thrun, S., Burgard, W., & Fox, D. ( 2005). Probabilistic robotics. Cambridge, MA: MIT Press.
79.
Tolman, E.C. ( 1932). Purposive behavior in animals and men. New York: Appleton.
80.
Toussaint, M. ( 2004). Learning a world model and planning with a self-organizing, dynamic neural system. In B. S. S. Thrun & L. K. Saul (Eds.), Advances in neural information processing systems (Vol. 16, pp. 929-936). Cambridge, MA: MIT Press.
81.
Toussaint, M. ( 2006). A sensorimotor map: Modulating lateral interactions for anticipation and planning. Neural Computation, 18, 1132-1155.
82.
Venturini, G. ( 1994). Adaptation in dynamic environments through a minimal probability of exploration. From Animals to Animats 3: Proceedings of the 3rd International Conference on Simulation of Adaptive Behavior (pp. 371-381).
83.
Wiemer, J.C. ( 2003). The time-organized map algorithm: Extending the self-organizing map to spatiotemporal signals. Neural Computation , 15(5), 1143-1171.
84.
Wiener, S.I., Berthoz, A., & Zugaro, M.B. ( 2002). Multisensory processing in the elaboration of place and head direction responses by limbic system neurons. Cognitive Brain Research, 14, 75-90.
85.
Wolpert, D.M. & Kawato, M. ( 1998). Multiple paired forward and inverse models for motor control. Neural Networks, 11, 1317-1329.
