Evolution and Learning: Evolving Sensors in a Simple MDP Environment

Abstract

Natural intelligence and autonomous agents face difficulties when acting in information-dense environments. Assailed by a multitude of stimuli they have to make sense of the inflow of information, filtering and processing what is necessary, but discarding that which is unimportant. This paper aims at investigating the interactions between evolution of the sensorial channel extracting the information from the environment and the simultaneous individual adaptation of agent-control. Our particular goal is to study the influence of learning on the evolution of sensors, with learning duration being the tunable parameter. A genetic algorithm governs the evolution of sensors appropriate for the agent solving a simple grid world task. The performance of the agent is taken as fitness; ‘sensors’ are conceived as a map from environmental states to agent observations, and individual adaptation is modeled by Q-learning. Our experimental results show that due to the principles of cognitive economy learning and varying the degree thereof actually transforms the fitness landscape. In particular we identify a trade-off between learning speed (load) and sensor accuracy (error). These results are further reinforced by theoretical analysis: we derive an analytical measure for the quality of sensors based on the mutual entropy between the system of states and the selection of an optimal action, a concept recently proposed by Polani, Martinetz, and Kim.

Get full access to this article

View all access options for this article.

References

Ackley, D., & Littman, M. (1991). Interactions between learning and evolution. In C. G. Langton, C. Taylor, J. D. Farmer, & S. Rasmussen (Eds.), Artificial life II, SFI studies in the sciences of complexity (Vol. X, pp. 487-509). Reading, MA: Addison-Wesley .

Adami, C. (1998). Introduction to artificial life. New York Springer Verlag .

Barlow, H. B. (1959). Sensory mechanisms, the reduction of redundancy, and intelligence . In Proceedings of the Symposium on the Mechanisation of Thought Processes, (pp. 535-539 ).

Bellman, R. (1957). Dynamic programming. NJ Princeton University Press .

Bruner, J. S., Goodnow, J. J., & Austin, G. A. (1956). A study of thinking. NY Wiley and Sons .

Dautenhahn, K., Polani, D., & Uthmann, T. (Eds.). (2001). Special issue on sensor evolution. Cambridge, MA: MIT Press .

Harvey, I., Husbands, P., & Cliff, D. (1994). Seeing the light: artificial evolution, real vision . In D. Cliff, P. Husbands, J.-A. Meyer, & S.Wilson (Eds.), From animals to animats 3, Proceedings of the 3rd International Conference on Simulation of Adaptive Behavior, SAB94 (pp. 392-401 ). Boston, MA: MIT Press/Bradford Books.

Hinton, G. E., & Nowlan, S. J. (1987). How learning can guide evolution . Complex Systems, 1(1), 495-502 .

Kaelbling, L. P., Littman, M. L., & Moore, A. P. (1996). Reinforcement learning: A survey . Journal of Artificial Intelligence Research, 4, 237-285 .

10.

Kortmann, R., & Herik, E. P. J. van den. (2001). Evolution of visual resolution constrained by a trade-off . Artificial Life (Special Issue on Sensor Evolution), 7(2), 125-145 .

11.

Lee, W. P., Hallam, J., & Lund, H. (1996). A hybrid GP/GA aproach for co-evolving controllers and robot bodies to achieve fitness-specified tasks . In Proceedings IEEE 3rd International Conference on Evolutionary Computation. NJ: IEEE Press.

12.

Liese, A., Polani, D., & Uthmann, T. (2001). Study of the simulated evolution of the spectral sensitivity of visual agent receptors . Artificial Life (Special Issue on Sensor Evolution), 7(2), 99-124 .

13.

Mark, A., Polani, D., & Uthmann, T. (1998). A framework for Sensor Evolution in a population of Braitenberg vehicle-like agents . In C. Adami, R. B. H. Kitano, & C. Taylor (Eds.), Proceedings of Artificial Life IV. Cambridge, MA: MIT Press.

14.

Maylay, G. (1996). Landscapes, learning costs, and genetic assimilation . Evolution, Learning and Instinct: 100 Years of the Baldwin Effect. A Special Edition of Evolutionary Computation, 4(3).

15.

Menczer, F., & Belew, R. (1994). Evolving sensors in environments of controlled complexity. In R. Brooks & P. Maes (Eds.), Artificial life IV. Cambridge, MA: MIT Press .

16.

Morrin, R., Forin, B., & Archer, W. (1961). Information processing behavior, the role of irrelevant stimulus information . Journal of Experimental Psychology, 61, 89-96 .

17.

Nehaniv, C. L. (1999). Meaning for observers and agents . In Proceedings IEEE International Symposium on Intelligent Control/Intelligent Systems and semiotics.

18.

Nolfi, S., Elman, J., & Parisi, D. (1994). Learning and evolution in neural networks . Adaptive Behavior, 3, 5-28 .

19.

Nolfi, S., & Floreano, D. (1999). Learning and evolution . Autonomous Robots, 7(1).

20.

Nolfi, S., & Floreano, D. (2000). Evolutionary robotics—the biology, intelligence, and technology of self-organizing machines. Cambridge, MA: MIT Press .

21.

Polani, D., Martinetz, T., & Kim, J. (2001). An information-theoretic approach for the quantification of relevance . In J. Kelemen & P. Sosik (Eds.), Proceedings 6th European Conference on Artificial Life. Berlin: Springer Verlag.

22.

Rosch, E. (1978). Principles of categorization. In E. Rosch & B. B. Lloyd (Eds.), Cognition and categorization. Hillsdale, New Jersey: Lawrence Erlbaum .

23.

Singh, S., Jaakkola, T., & Jordan, M. I. (1994). Learning without state-estimation in partially observable markovian decision processes . Proceedings of the 11th Machine Learning Conference.

24.

Sutton, R., & Barto, A. (1998). Reinforcement learning: An introduction. MIT Press .

25.

Tishby, N., Pereira, F., & Bialek., W. (1999). The information bottleneck method . In Proceedings of the 37th Annual Allerton Conference on Communication, Control, and Computing. Illinois.

26.

Todd, P. M., & Miller, G. F. (1991). Exploring adaptive agency II: Simulating the evolution of associative learning . In J. A. Meyer & S. W. Wilson (Eds.), From animals to animats. Proceedings of the First International Conference on Simulation of Adaptive Behavior. Cambridge, MA: MIT Press.

27.

Turney, P., Whitley, D., & Anderson, R. (1996). Evolution, learning, and instinct: 100 years of the baldwin effect . Spec. Issue of Evol. Comp. on the Baldwin Effect, 4.

28.

Watkins, C. J. C. H. (1989). Learning from delayed rewards. Unpublished doctoral dissertation, Cambridge, England.

29.

Watkins, C. J. C. H., & Dayan, P. (1992). Q-learning . Machine Learning, 8(3), 279-292 .

30.

Whitehead, S. D., & Ballard, D. H. (1991). Learning to perceive and act by trial and error . Machine Learning, 7, 45-83 .

31.

Wittgenstein, L. (1953). Philosophical investigations. Macmillan, New York .