Emotion plays an important role in thinking. In this article we study affective control of the amount of simulated anticipatory behavior in adaptive agents using a computational model. Our approach is based on model-based reinforcement learning (RL) and inspired by the simulation hypothesis (Cotterill, 2001; Hesslow, 2002). The simulation hypothesis states that thinking is internal simulation of behavior using the same sensory-motor systems as those used for overt behavior. Here, we study the adaptiveness of an artificial agent, when action-selection bias is induced by an affect-controlled amount of simulated anticipatory behavior . To this end, we introduce an affect-controlled simulation-selection mechanism that uses the predictions of the agent's RL model to select anticipatory behaviors for simulation. Based on eXperiments with adaptive agents in two nondeterministic partially observable grid-worlds we conclude that (1) internal simulation has an adaptive benefit and (2) affective control can reduce the amount of simulation needed for this benefit. This is specifically the case if the following relation holds: positive affect decreases the amount of simulation towards simulating the best potential neXt action, while negative affect increases the amount of simulation towards simulating all potential neXt actions. In essence we use artificial affect to control mental eXploration versus eXploitations. Thus, agents “feeling positive” can think ahead in a narrow sense and free up working memory resources, while agents “feeling negative” must think ahead in a broad sense and maXimize usage of working memory. Our results are consistent with several psychological findings on the relation between affect and learning, and contribute to answering the question of when positive versus negative affect is useful during adaptation.
Ashby, F.G., Isen, A.M., & Turken, U. (1999). A neuropsychological theory of positive affect and its influence on cognition. Psychological Review, 106, 529—550.
2.
Avila-Garcia, O., & Cañamero, L. (2004). Using hormonal feedback to modulate action selection in a competitive scenario. From Animals to Animats 8: Proceedings of the 8th International Conference on Simulation of Adaptive Behavior (pp. 243—252). Cambridge, MA: MIT Press.
3.
Aylett, R. (2006). Emotion as an integrative process between non-symbolic and symbolic systems in intelligent agents. Proceedings of the AISB'06 Symposium on Architecture of Brain and Mind (pp. 43—47). AISB Press.
4.
Baddeley, A. (2000). The episodic buffer: A new component of working memory? Trends in Cognitive Sciences, 4, 417—423.
5.
Belavkin, R.V. (2004). On relation between emotion and entropy. Proceedings of the AISB'04 Symposium on Emotion, Cognition and A fective Computing (pp. 1—8). AISB Press.
6.
Berridge, K.C. (2003). Pleasures of the brain. Brain andCognition, 52, 106—128.
7.
Bickhard, M.H. (1998). Levels of representationality. Journal of Experimental and Theoretical Artificial Intelligence, 10, 179—215.
8.
Blanchard, A.J., & Cañamero, L. (2006). Modulation of exploratory behavior for adaptation to the context. Proceedings of the AISB'06 Symposium on Biologically Inspired Robotics (Biro-net) (pp 131—137). AISB Press.
9.
Botelho, L.M., & Coelho, H. (1998). Information processing, motivation and decision making. Proceedings of the 4th International Workshop on Artificial Intelligence in Economics and Management, (pp. 233—250).
10.
Broekens, J. (2005). Internal simulation of behavior has an adaptive advantage. Proceedings of the CogSci'05 Conference (pp. 342—347). Mahwah, NJ: Lawrence Erlbaum Associates.
11.
Broekens, J., & DeGroot, D. (2004). Emergent representations and reasoning in adaptive agents. Proceedings of the Third International Conference on Machine Learning and Applications (pp. 207—214). Los Alamitos, CA: IEEE Press.
12.
Broekens, J., Kosters, W.A., & Verbeek, F.J. (2007). On affect and self-adaptation: Potential benefits of valence-controlled action-selection. In: J. Mira and J.R. Álvarez (Eds.), IWINAC 2007, Part I, LNCS 4527 (pp. 357—366). Berlin : Springer.
13.
Broekens, J., & Verbeek, F.J. (2005). Simulation, emotion and information processing: Computational investigations of the regulative role of pleasure in adaptive behavior. Proceedings of the Workshop on Modeling Natural Action Selection (pp. 166—173). AISB Press.
14.
Butz, M.V., Sigaud, O., & Gerard, P. (2003). Internal models and anticipations in adaptive learning systems. In: LNAI 2684: Anticipatory behavior in adaptive learning systems (pp. 86—109). Berlin: Springer.
15.
Cañamero, D. (2000). Designing emotions for activity selection. Department of Computer Science Technical Report DAIMI PB 545. University of Aarhus, Denmark.
16.
Clore, G.L., & Gasper, K. (2000). Feeling is believing: Some affective influences on belief. In: N. Frijda, A. S. R. Manstead, & S. Bem (Eds.), Emotions and beliefs. Cambridge, UK: Cambridge University Press.
Cos-Aguilera, I., Cañamero, L., Hayes, G.M., & Gillies, A. (2005). Ecological integration of affordances and drives for behaviour selection. Proceedings of the Workshop on Modeling Natural Action Selection, Edinburgh: AISB Press, (pp. 225—228).
19.
Cotterill, R.M.J. (2001). Cooperation of the basal ganglia, cerebellum, sensory cerebrum and hippocampus: Possible implications for cognition, consciousness, intelligence and creativity. Progress in Neurobiology, 64, 1—33.
20.
Craig, S.D., Graesser, A.C., Sullins, J., & Gholson, B. (2004). Affect and learning: An exploratory look into the role of affect in learning with AutoTutor. Journal of Educational Media, 29, 241—250.
21.
Custers, R., & Aarts, H. (2005). Positive affect as implicit motivator: On the nonconscious operation of behavioral goals. Journal of Personality and Social Psychology, 89, 129—142.
22.
Damasio, A.R. (1994). Descartes' error: Emotion, reason, and the human brain. New York: PenguinPutnam.
Deheane, S., Sergent, C., & Changeux, J-P. (2003). A neuronal network model linking subjective reports and objective physiological data during conscious perception. Proceedings of the Natural Academy of Sciences , 100, 8520—8525.
26.
Demiris, Y., & Johnson, M. (2003). Distributed, predictive perception of actions: A biologically inspired robotics architecture for imitation and learning. ConnectionScience, 15, 231—243.
27.
Doya, K. (2002). Metalearning and neuromodulation. Neural Networks, 15, 495—506.
28.
Dreisbach, G., & Goschke, K. (2004). How positive affect modulates cognitive control: Reduced perseveration at the cost of increased distractibility. Journal of Experimental Psychology: Learning, Memory, and Cognition, 30, 343—353.
29.
Forgas, J.P. (2000). Feeling is believing? The role of processing strategies in mediating affective influences in beliefs. In: N. Frijda, A. S. R. Manstead, & S. Bem (Eds.), Emotions and beliefs. Cambridge, UK: Cambridge University Press.
30.
Foster, D.J., & Wilson, M.A. (2006). Reverse replay of behavioural sequences in hippocampal place cells during the awake state. Nature, 440, 680—683.
31.
Frijda, N.H., Manstead, A.S.R., & Bem. S. (2000). The influence of emotions on beliefs. In: N. Frijda, A. S. R. Manstead, & S. Bem (Eds.), Emotions and beliefs. Cambridge, UK: Cambridge University Press.
32.
Frijda, N.H., & Mesquita, B. (2000). Beliefs through Emotions. In: N. Frijda, A. S. R. Manstead , & S., Bem, (Eds.), Emotions and beliefs. Cambridge, UK: Cambridge University Press.
33.
Gadanho, S.C. (2003). Learning behavior-selection by emotions and cognition in a multi-goal robot task. Journal of Machine Learning Research , 4, 385—412.
34.
Griffith, P.E. (1999). Modularity & the psychoevolutionary theory of emotion. Mind and Cognition: An Anthology. Oxford: Blackwell.
35.
Hesslow, G. (2002). Conscious thought as simulation of behaviour and perception. Trends in Cognitive Sciences, 6, 242—247.
36.
Hoffmann, H., & Möller, R. (2004). Action selection and mental transformation based on a chain of forward models. Proceedings of the 8th International Conference on the Simulation of Adaptive Behavior (pp. 213—222). Cambridge, MA: MIT Press.
37.
Hogewoning, E., Broekens, J., Eggermont, J., & Bovenkamp, E.G.P. (2007). Strategies for affect-controlled action-selection in Soar-RL. In: J. Mira and J.R. Álvarez (Eds.), IWINAC 2007, Part II, LNCS 4528 (pp. 501—510). Berlin: Springer.
38.
Kaelbling, L.P., Littman, M.L., & Moore, A.W. (1996). Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4, 237—285.
39.
Lahnstein, M. (2005). The emotive episode is a composition of anticipatory and reactive evaluations. Proceedings of the AISB'05 Symposium on Agents that Want and Like (pp. 62—69). AISB Press .
40.
McCallum, A. (1995). Instance-based utile distinctions for reinforcement learning with hidden state. Proceedings of the Twelfth International Conference on Machine Learning, San Francisco: Morgan Kaufmann, (pp. 387—395).
41.
McMahon, A., Scott, D., Baxter, P., & Browne, W. (2006). An autonomous explore/exploit strategy. Proceedings of the AISB'06 Symposium on Nature Inspired Systems (pp. 192— 201). AISB Press.
42.
Montague, P.R., Hyman S.E., & Cohen, J.D. (2004). Computational roles for dopamine in behavioural control. Nature, 431, 760—767.
Phaf, R.H., & Rotteveel, M. (2005). Affective modulation of recognition bias. Emotion, 5, 309—318.
45.
Picard, R.W., Papert, S., Bender, W., Blumberg, B., Breazeal, C. Cavallo, D., Machover, T., Resnick, M., Roy, D., & Strohecker , C. (2004). Affective learning—A manifesto. BT Technology Journal , 22, 253—269.
46.
Rolls, E.T. (2000). Précis of The brain and emotion. Behavioral and Brain Sciences, 23, 177—191.
47.
Rose, S.A., Futterweit, L.R., & Jankowski, J.J. (1999). The relation of affect to attention and learning in infancy. Child Development, 70, 549—559.
48.
Russell, J.A. (2003). Core affect and the psychological construction of emotion. Psychological Review, 110, 145—172.
49.
Salichs, M.A., & Malfaz, M. (2006). Using emotions on autonomous agents. The role of happiness, sadness and fear. Proceedings of the AISB'06 Symposium on Integrative Approaches to Machine Consciousness (pp. 157—164). AISB Press.
50.
Scherer, K.R. (2001). Appraisal considered as a process of multilevel sequential checking. In: K. R. Scherer, A. Schott, & T. Johnstone (Eds.) Appraisal processes in emotion: Theory, Methods, Research. New York: Oxford Univ. Press,.
51.
Schultz, W., Dayan, P., & Montague, P.R. (1997). A neural substrate of prediction and reward. Science, 275, 1593—1599.
52.
Schweighofer, N., & Doya, K. (2003). Meta-learning in reinforcement learning. Neural Networks, 16, 5—9.
53.
Shanahan, M. (2006). A cognitive architecture that combines internal simulation with a global workspace. Consciousness and Cognition , 15, 433—449.
54.
Sutton, R.S. (1990). Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. Proceedings of the Seventh International Conference on Machine Learning (pp. 216—224). Morgan Kaufmann.
55.
Sutton, R.S., & Barto, A.G. (1998). Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press.
56.
Tyrell, T. (1993). Computational mechanisms for action selection. Ph.D. Thesis, University of Edinburgh.
57.
van Dartel, M.F., & Postma, E.O. (2005) Symbol manipulation by internal simulation of perception and behaviour. Proceedings of the 5th International Workshop on Epigenetic Robotics. Lund University Cognitive Studies Vol. 123 (pp. 121—124). Lund: Lund University.
58.
van Dartel, M., Postma, E. & van den Herik, J. (2004) Categorisation through internal simulation of perception and behaviour. Proceedings of the 16th Belgium-Netherlands Conference on Artificial Intelligence (pp. 187—194).
59.
von Hecker, U., & Meiser, T., (2005). Defocused attention in depressed mood: Evidence from source monitoring. Emotion, 5, 456—463.
60.
Velasquez, J.D. (1998). A computational framework for emotion-based control . In: SAB'98 Work-shop on Grounding Emotions in Adaptive Systems, 5th International Conference of the Society for Adaptive Behavior, Zurich.
61.
Ziemke, T., Jirenhed, D., & Hesslow, G. (2005). Internal simulation of perception: A minimal neuro-robotic model. Neurocomputing, 68, 85—104.
