The aim of the Cyber Rodent project is to understand the origins of our reward and affective systems by building artificial agents that share the same intrinsic constraints as natural agents: Self-preserva tion and self-reproduction. A Cyber Rodent is a robot that can search for and recharge from battery packs on the floor and copy its programs to a nearby agent through its infrared communication port. This article reviews our research topics so far, including (1) evolution of neural controllers, (2) learning of foraging and mating behaviors, (3) evolution of learning architectures and meta-parameters, (4) simultaneous learning of multiple agents in a body, and (5) learning and evolution in a self-sustained colony. We discuss our future directions and expected contributions.
Birk, A. (1998). Robot learning and self-sufficiency: What the energy-level can tell us about a robot’s performance. In A. Birk & J. Demiris (Eds.), Learning Robots, Lecture Notes in Artificial Intelligence, Vol. 1545. Berlin: Springer .
2.
Capi, G. , & Doya, K. (2003). Evolving recurrent neural controllers for sequential tasks: A parallel implementation. In R. Sarker et al. (Eds.), Proceedings of Congress on Evolutionary Computation: Vol. 1 (pp. 514–519). IEEE Press .
3.
Capi, G. , & Doya, K. (2005). Evolution of neural architecture fitting environmental dynamics . Adaptive Behavior, 13, 53–66 .
4.
Capi, G. , Uchibe, E. , & Doya, K. (2002). Selection of neural architecture and the environment complexity. In T. Asakura , and K. Murase (Eds.), Proceedings of the Third International Symposium on Human and Artificial Intelligent Systems (pp. 231–237). Japan: Fuki University Press .
5.
Capi, G. , Uchibe, E. , & Doya, K. (2003). Selection of neural architecture and the environment complexity. In K. Murase , and T. Asakura (Eds.), Dynamic Systems Approach for Embodiment and Sociality From Ecological Psychology to Robotics 6 (pp. 311–317). Adelaide: Advanced Knowledge International .
6.
Dietterich, T. G. (2000). Hierarchical reinforcement learning with the MAXQ value function decomposition . Journal of Artificial Intelligence Research, 13, 227–303 .
7.
Doncieux, S. , & Meyer, J.-A. (2004). Evolving modular neural networks to solve challenging control problems. In Proceedings of the Fourth International ICSC Symposium on Engineering of Intelligent Systems (EIS 2004). Canada: ICSC Academic Press .
8.
Doya, K. (2000). Complementary roles of basal ganglia and cerebellum in learning and motor control . Current Opinion in Neurobiology, 10, 732–739 .
9.
Doya, K. (2002). Metalearning and neuromodulation . Neural Networks, 15, 495–506 .
10.
Doya, K. (2003). Cyber rodents: Self-preserving, self-reproducing robotic colony. Technical report , ATR Up to Date, 5, 12–13 .
11.
Elfwing, S. , Uchibe, E. , & Doya, K. (2003). An evolutionary approach to automatic construction of the structure in hierarchical reinforcement learning . In Proceedings of Genetic and Evolutionary Computation Conference (GECCO2003) (pp. 507–509 ). Springer.
12.
Elfwing, S. , Uchibe, E. , Doya, K. , & Christensen, H. I. (2004). Multi-agent reinforcement learning: Using macro actions to learn a mating task . In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS2004) (pp. 3164–3169 ). IEEE Press.
13.
Eriksson, A. , Capi, G. , & Doya, K. (2003). Evolution of metaparameters in reinforcement learning algorithm . In Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS2003) (pp. 412–417 ). IEEE Press.
14.
Mataric, M. , & Cliff, D. (1996). Challenges in evolving controllers for physical robots . Robotics and Autonomous Systems, 19(1), 67–83 .
15.
McFarland, D. , & Spier, E. (1997). Basic cycles, utility and opportunism in self-sufficient robots . Robotics and Autonomous Systems, 20, 179–190 .
16.
Miyazaki, K. , Miyazaki, K. W. , & Matsumoto, G. (2004). Different representation of forthcoming reward in nucleus accumbens and medial prefrontal cortex . Neuroreport, 15, 721–726 .
17.
Nehmzow, U. (2002). Physically embedded genetic algorithm learning in multi-robot scenarios: The PEGA algorithm. In C. G. Prince , Y. Demiris , Y. Maram , H. Kozima , and C. Balkenius (Eds.), Second International Workshop on Epigenetic Robotics: Modelling Cognitive Development in Robotic Systems.
18.
Ng, A.Y. , Harada, D. , & Russell, S. (1999). Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the Sixteenth International Conference on Machine Learning (pp. 278–287). San Francisco: Morgan Kaufmann .
19.
Niv, Y. , Joel, D. , Meilijson, I. , & Ruppin, E. (2002). Evolution of reinforcement learning in uncertain environments: A simple explanation for complex foraging behaviors . Adaptive Behavior, 10(1), 5–24 .
20.
Ruppin, E. (2002). Evolutionary autonomous agents: A neuroscience perspective . Nature Review Neuroscience, 3, 132–141 .
21.
Schultz, W. , Dayan, P. , & Montague, P. R. (1997). A neural substrate of prediction and reward . Science, 275, 1593–1599 .
22.
Spier, E. , & McFarland, D. (1996). A finer-grained motivational model of behaviour sequencing. In P. Maes , M. Mataric , J.-A. Meyer , J. Pollack , and S. W. Wilson (Eds.), Proceedings of the Fourth International Conference on Simulation of Adaptive Behavior (pp. 255–263). Cambridge, MA: MIT Press .
23.
Sutton, R. , & Barto, A. G. (1998). Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press/Bradford Books .
24.
Sutton, R. , Precup, D. , & Singh, S. (1999). Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning . Artificial Intelligence, 112, 181–211 .
25.
Tanaka, S. , Doya, K. , Okada, G. , Ueda, K. , Okamoto, Y. , & Yamawaki, S. (2004). Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops . Nature Neuroscience, 7, 887–893 .
26.
Uchibe, E. , & Doya, K. (2004). Competitive-cooperative-concurrent reinforcement learning with importance sampling . In Proceedings of International Conference on Simulation of Adaptive Behavior: From animals and animats (SAB2004) (pp. 287–296 ).
27.
Usher, M. , Cohen, J. D. , Servan-Schreiber, D. , Rajkowski, J. , & Aston-Jones, G. (1999). The role of locus coeruleus in the regulation of cognitive performance . Science, 283, 549–554 .
28.
Usui, Y. , & Arita, T. (2003). Situated and embodied evolution in collective evolutionary robotics. In M. Sugisaka , and H. Tanaka (Eds.), Proceedings of the Seventh International Symposium on Artificial Life and Robotics (pp. 487–490).
29.
Walker, J. , Garrett, S. , & Wilson, W. (2003). Evolving controllers for real robots: A survey of the literature . Adaptive Behavior, 11(3), 179–203 .
30.
Watson, R. A. , Ficici, S. G. , & Pollack, J. B. (2002). Embodied evolution: Distributing an evolutionary algorithm in a population of robots . Robotics and Autonomous Systems, 39, 1–18 .
