Several researchers have demonstrated how complex action sequences can be learned through neuroevolution (i.e., evolving neural networks with genetic algorithms). However, complex general behavior such as evading predators or avoiding obstacles, which is not tied to specific environments, turns out to be very difficult to evolve. Often the system discovers mechanical strategies, such as moving back and forth, that help the agent cope but are not very effective, do not appear believable, and do not generalize to new environments. The problem is that a general strategy is too difficult for the evolution system to discover directly. This article proposes an approach wherein such complex general behavior is learned incrementally, by starting with simpler behavior and gradually making the task more challenging and general. The task transitions are implemented through successive stages of Delta coding (i.e., evolving modifications), which allows even converged populations to adapt to the new task. The method is tested in the stochastic, dynamic task of prey capture and is compared with direct evolution. The incremental approach evolves more effective and more general behavior and should also scale up to harder tasks.
Anderson, C.W. (1989). Learning to control an inverted pendulum using neural networks. IEEE Control Systems Magazine, 9, 31-37.
2.
Barto, A.G., Sutton, R.S., & Anderson, C.W. (1983). Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics, SMC-13, 834-846.
3.
Brooks, R.A. (1986). A robust layered control system for a mobile robot . IEEE Journal of Robotics and Automation, 2(10), 14-23.
4.
Cliff, D., Harvey, I., & Husbands, P. (1992). Incremental evolution of neural network architectures for adaptive behavior (Tech. Rep. No. CSRP256). Brighton , UK: School of Cognitive and Computing Sciences, University of Sussex.
5.
Colombetti, M., & Dorigo, M. (1992a). Learning to control an autonomous robot by distributed genetic algorithms. In J.-A. Meyer, H. L. Roitblat & S. W Wilson (Eds.), From animals to animats 2: Proceedings of the Second International Conference on Simulation of Adaptive Behavior. Cambridge, MA: MIT Press.
6.
Colombetti, M., & Dorigo, M. (1992b). Robot shaping: Developing situated agents through learning (Tech. Rep. No. TR-92-040). Berkeley, CA: International Computer Science Institute.
7.
Elman, J.L. (1991). Incremental learning, or the importance of starting small. In Proceedings of the Thirteenth Annual Conference of the Cognitive Science Society. Hillsdale, NJ: Lawrence Associates.
8.
Law, D., & Miikkulainen, R. (1994). Grounding robotic control with genetic neural networks (Tech. Rep. No. AI93-223). Austin: Department of Computer Sciences, The University of Texas at Austin.
9.
Lin, L.-J. (1992). Self-improving reactive agents based on reinforcement learning, planning, and teaching. Machine Learning, 8, 293-321.
10.
Lin, L.J. (1993). Hierarchical learning of robot skills by reinforcement. In Proceedings of the International Joint Conference on Neural NetworksBoston: Kluwer Academic.
11.
Mataric, M., & Cliff, D. (1995). Challenges in evolving controllers for physical robots (Tech. Rep. No. CS-95-184). Waltham, MA: Brandeis University, Department of Computer Science .
12.
Miller, G., & Cliff, D. (1994). Co-evolution of pursuit and evasion. I: Biological and game-theoretic foundations (Tech. Rep. No. CSRP311). Brighton, UK: School of Cognitive and Computing Sciences, University of Sussex.
13.
Moriarty, D.E. (1997). Symbiotic evolution of neural networks in sequential decision tasks (Tech. Rep. No. UT-AI97-259). [Doctoral thesis.]Austin: Department of Computer Sciences, The University of Texas at Austin.
14.
Moriarty, D.E., & Miikkulainen, R. (1996a). Efficient reinforcement learning through symbiotic evolution. Machine Learning, 22, 11-32.
15.
Moriarty, D.E., & Miikkulainen, R. (1996b). Evolving obstacle avoidance behavior in a robot arm. In P Maes, M. Mataric, J.-A. Meyer & J. Pollack (Eds.), From animals to animats: The Fourth International Conference on Simulation of Adaptive Behavior (SAB '96). Cambridge, MA: MIT Press.
16.
Nolfi, S., Elman, J.L., & Parisi, D. (1990). Learning and evolution in neural networks (Tech. Rep. No. 9019). San Diego: Center for Research in Language, University of California.
17.
Nolfi, S., & Parisi, D. (1994). Good teaching inputs do not correspond to desired responses in ecological neural networks. Neural Processing Letters, 1(2), 1-4.
18.
Nolfi, S., & Parisi, D. (1995). Learning to adapt to changing environments in evolving neural networks (Tech. Rep. No. 95-15). Rome, Italy: Institute of Psychology, National Research Council.
19.
Pendrith, M. (1994). On reinforcement learning of control actions in noisy and non-Markovian domains (Tech. Rep. UNSW-CSE-TR-9410). Sydney, Australia: School of Computer Science and Engineering, The University of New South Wales.
20.
Perkins, S., & Hayes, G. (1996). Robot shaping—principles, methods, and architectures (Tech. Rep. No. 795). Edinburgh: Department of Artificial Intelligence, University of Edinburgh.
21.
Ring, M.B. (1994). Continual learning in reinforcement environments. Unpublished doctoral thesis, Department of Computer Sciences, The University of Texas at Austin, Austin, TX.
22.
Saravanan, N., and Fogel, D.B. (1995). Evolving neural control systems. IEEE Expert, 10(3), 23-27.
23.
Singh, S. (1992). Transfer of learning by composing solutions of elemental sequential tasks. Machine Learning, 8, 323-339.
24.
Thrun, S. (1996). Explanation-based neural network learning: A lifelong learning approach. Amsterdam: Kluwer Academic.
Whitley, D., & Kauth, J. (1988). Genitor: A different genetic algorithm. In Proceedings of the 1988 Rocky Mountain Conference on Artificial Intelligence. Fort Collins, CO: Computer Science Department, Colorado State University.
27.
Whitley, D., Mathias, K., & Fitzhorn, P. (1991). Delta-coding: An iterative search strategy for genetic algorithms. In Proceedings of the Fourth International Conference on Genetic AlgorithmsLos Altos, CA: Morgan Kaufmann.
28.
Wieland, A. (1990). Evolving controls for unstable systems. In D. S. Touretzky, J. L. Elman & T. J. Sejnowski (Eds.), Proceedings of the 1990 Connectionist Models Summer School. San Mateo, CA: Morgan Kaufmann .
29.
Wieland, A. (1991). Evolving neural network controllers for unstable systems. In Proceedings of the International Joint Conference on Neural Networks (Seattle, WA), Vol. 2. Piscataway, NJ: IEEE.
30.
Wilson, S.W. (1991). The animat path to AI. In J. Meyer & S. Wilson (Eds.), Proceedings of the First International Conference on Simulation of Adaptive Behavior. Cambridge, MA: MIT Press/Bradford Books.
31.
Yamauchi, B., & Beer, R. (1994). Integrating reactive, sequential, and learning behavior using dynamical neural networks. In D. Cliff, P. Husbands, J.-A. Meyer & S. Wilson (Eds.), From animals to animals 3: Proceedings of the Third International Conference on Simulation of Adaptive Behavior. Cambridge, MA: MIT Press/Bradford Books .
