In this article we propose a framework for performing embodied evolution with a limited number of robots, by utilizing time-sharing in subpopulations of virtual agents hosted in each robot. Within this framework, we explore the combination of within-generation learning of basic survival behaviors by reinforcement learning, and evolutionary adaptations over the generations of the basic behavior selection policy, the reward functions, and metaparameters for reinforcement learning. We apply a biologically inspired selection scheme, in which there is no explicit communication of the individuals’ fitness information. The individuals can only reproduce offspring by mating—a pair-wise exchange of genotypes—and the probability that an individual reproduces offspring in its own subpopulation is dependent on the individual’s ‘‘health,’’ that is, energy level, at the mating occasion. We validate the proposed method by comparing it with evolution using standard centralized selection, in simulation, and by transferring the obtained solutions to hardware using two real robots.
Baldwin, J. ( 1896). A new factor in evolution. American Naturalist , 30, 441-451.
2.
Doya, K. ( 2002). Metalearning and neuromodulation. Neural Networks, 15(4), 485-506.
3.
Doya, K., & Uchibe, E. ( 2005). The Cyber Rodent project: Exploration of adaptive mechanisms for self-preservation and self-reproduction. Adaptive Behavior , 13(2), 149-160.
4.
Elfwing, S., Uchibe, U., Doya, K., & Christensen, H.I. (2005). Biologically inspired embodied evolution of survival. In Proceedings of the IEEE Congress on Evolutionary Computation (CEC2005) (Vol. 3)IEEE Press, Piscataway, NJ (pp. 2210-2216).
5.
Elfwing, S., Uchibe, U., Doya, K., & Christensen, H. (2007). Evolutionary development of hierarchical learning structures. IEEE Transactions on Evolutionary Computation, 11(2), 249-264.
6.
Elfwing, S., Uchibe, U., Doya, K., & Christensen, H. (2008). Coevolution of shaping rewards and meta-parameters in reinforcement learning. Adaptive Behavior, 16(6), 400-412.
7.
Eriksson, A., Capi, G., & Doya, K. ( 2003). Evolution of meta-parameters in reinforcement learning algorithm. In C. S. G. Lee & J. Yuh (Eds.), Proceedings of the IEEE/RSJ Conference on Intelligent Robots and Systems (IROS2003) (pp. 412-417). Piscataway, NJ: IEEE Press.
8.
Floreano, D., & Mondada, F. ( 1996). Evolution of plastic neurocontrollers for situated agents . In P. Maes, M. Mataric, J.-A. Meyer, J. Pollack & S. Wilson (Eds.), Proceedings of the International Conference on Simulation of Adaptive Behavior (SAB1996) (pp. 401-410).
9.
Hinton, G., & Nowlan, S. ( 1987). How learning can guide evolution. Complex Systems, 1, 495-502.
10.
Laud, A., & DeJong, G. (2003). The influence of reward on the speed of reinforcement learning: An analysis of shaping. In T. Fawcett & N. Mishra (Eds.), Proceedings of the International Conference on Machine learning (ICML2003) (pp. 440-447).
11.
Nehmzow, U. ( 2002). Physically embedded genetic algorithm learning in multi-robot scenarios: The PEGA algorithm. In C. G. Prince, Y. Demiris, Y. Marom, H. Kozima & C. Balkenius (Eds.), Proceedings of the International Workshop on Epigenetic Robotics and Robotics (EPIROB2002) (pp. 115-123).
12.
Ng, A.Y., Harada, D., & Russell, S.J. ( 1999). Policy invariance under reward transformations: theory and application to reward shaping. In I. Bratko & S. Dzeroski (Eds.), Proceedings of the International Conference on Machine learning (ICML1999) (pp. 278-287). San Francisco, CA: Morgan Kaufmann Publishers Inc.
13.
Niv, Y., Joel, D., Meilijson, I., & Ruppin, E. ( 2002). Evolution of reinforcement learning in uncertain environments: A simple explanation for complex foraging behaviors. Adaptive Behavior, 10(1), 5-24.
14.
Nolfi, S., & Floreano, D. ( 2000). Evolutionary robotics. The biology, intelligence, and technology of self-organizing machines. Cambridge, MA: MIT Press.
15.
Rummery, G.A., & Niranjan, M. ( 1994). On-line Q-learning using connectionist systems. (Tech. Rep. CUED/F-INFENG/TR 166). Cambridge University Engineering Department.
16.
Ruppin, E. ( 2002). Evolutionary autonomous agents: A neuroscience perspective . Nature Review Neuroscience, 3, 132-141.
Sutton, R.S. ( 1996). Generalization in reinforcement learning: Successful examples using sparse coarse coding. In D. S. Touretzky, M. C. Mozer & M. E. Hasselmo (Eds.), Advances in neural information processing systems 8 (pp. 1038-1044). Cambridge, MA: MIT Press.
19.
Sutton, R.S., & Barto, A.G. ( 1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press.
20.
Turney, P., Whitley, D., & Anderson, R. ( 1996). Introduction to the special issue: Evolution, learning, and instinct: 100 years of the Baldwin effect. Evolutionary Computation, 4(3), iv-viii.
21.
Urzelai, J., & Floreano, D. ( 2000). Evolutionary robotics: Coping with environmental change . In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO2000) (pp. 941-948).
22.
Usui, Y., & Arita, T. ( 2003). Situated and embodied evolution in collective evolutionary robotics. In M. Sugisaka & H. Tanaka (Eds.), Proceedings of the International Symposium on Artificial Life and Robotics (AROB2003) (pp. 212-215).
23.
Watson, R., Ficici, S., & Pollack, J. ( 2002). Embodied evolution: Distributing an evolutionary algorithm in a population of robots. Robotics and Autonomous Systems , 39(1), 1-18.
24.
Whiteson, S., & Stone, P. ( 2006). Evolutionary function approximation for reinforcement learning. Journal of Machine Learning Research, 7, 877-917.
25.
Wiewiora, E. ( 2003). Potential-based shaping and Q-value initialization are equivalent. Journal of Artificial Intelligence Research, 19, 205-208.
