Plasticity in Value Systems and its Role in Adaptive Behavior

Abstract

Adaptive behavior requires the sensing of salient behavioral consequences which can act to modulate changes in neural connections linking sensory and motor structures. In previous work, we proposed that salient sensory events trigger neuronal value systems capable of modulating synaptic plasticity. Here, we investigate the capacity of value systems to modulate their own responses in the context of various conditioning tasks. To this end, we implement a modifiable value system incorporating anatomical and physiological properties within Darwin V, a neuronal model embedded in a mobile real world device. While exploring an environment containing stimulus objects, Darwin V's visual maps develop object-related neuronal responses. Phasic responses of a value system initially triggered only by object-"taste" (innate value) modulate changes in connections between visual and motor neurons, thus linking specific visual responses to appropriate motor outputs. Over time, Darwin V is able behaviorally to discriminate between "striped" objects with positive value (appetitive behavior) and objects with "blobs" with negative value (aversive behavior) based on vision alone. In parallel with modification of visuo-motor connections, value-dependent modification also occurs in connections from visual sensory maps to the value system itself. As a result, visual activity patterns become able directly to trigger value signals (acquired value). If acquired value is disabled, transfer of the value signal to stimuli preceding innately salient events does not occur, and behavioral responses due to aversive conditioning are subject to rapid extinction. If an auditory signal reliably precedes the visual appearance of an aversive object, Darwin V could be conditioned first to reject the object based on vision (primary conditioning), and subsequently based on sound alone (secondary conditioning). We compare the functional characteristics of value-dependent learning to formal notions of reinforcement learning. We suggest that plasticity in sensory afferents to value systems may provide a neurobiological basis for mediating the changing effects of saliency on adaptive behavioral responses.

Keywords

Value Plasticity Conditioning Reinforcement Categorization Vision

Get full access to this article

View all access options for this article.

References

Almássy, N. , Edelman, G.M. , and Sporns, O. (1998). Behavioral constraints in the developments of neuronal properties: A cortical model embedded in a real world device. Crebral Cortex, 8:346-361.

Almássy, N. and Sporns, O. (1998). Role of behavior in the development of complex neuronal properties. In Pfeifer, R. , Blumberg, B. , Meyer, J.-A. , and Wilson, S. W. , editors, From Animals to Animats 5, pages 311-320, Cambridge, MA. MIT Press.

Aston-Jones, G. and Bloom, F.E. (1981). Norepinephrine-containing locus coerulens neurons in behaving rats exhibit pronounced response to non-noxious environmental stimuli. Journal of Neuroscience, 1:887-900.

Aston-Jones, G. , Ennes. M. , Pieribone, V.A. , Nickell, W.T. , and Shipley, M.T. (1986). The brain nucleus locus coeruleus: restricted afferent control of a broad efferent network. Science, 234(4777) :734-715.

Aston-Jones, G. , Rajkowski, J. , and Kubiak, P. (1997). Conditioned responses of monkey locus coeruleus neurons anticipate acquisition of discriminative behavior in a vigilance task . Neuroscience, 80:697-715.

Aston-Jones, G. , Rajkowski, J. , Kubiak, P. , and Alexinsky, T. (1994). Locus coeruleus neurons in the monkey are selectively activated by attended cues in a vigilance task. Journal of Neuroscience, 14(7):4467-4480.

Aston-Jones, G. Segal, M. , and Bloom, F.E. (1980). Brain aminergic axons exhibit marked variability in conduction velocity. Brain Research, 195(1):215-222.

Balkenius, C. and Morén, J. (1998). Computational models of classical conditioning: A comparative study. In Pfeifer, R. , Blumberg, B. , Meyer, J.-A and Wilson, S.W. , editor, From Animals to Animats 5, pages 348-353, Cambridge, MA. MIT Press.

Barto, A.G. (1990). Connectionist learning for control : An overview . In Miller, T. , Sutton, R. S. , And Werbos, P J. , editor, Neural Networks for Control, pages 5-58, Cambridge, MA. MIT Press.

10.

Baskerville, K.A. , Schweitzer, J.B. , and Herron, P. (1997). Effects of cholinergic depletion on experience-dependent plasticity in the cortex of the rat. Neuroscience, 80(4):1159-1169.

11.

Bear, M.F. and Singer, W. (1986). Modulation of visual cortical plasticity by acetylcholine and noradrenaline. Nature, 320:172-176.

12.

Bienenstock, E.L. , Cooper, L.N. , and Munro, P. (1982). Theory for the development of neuron selectivity: Orientation selectivity and binocular interaction in visual cortex. Journal ofNeuroscience, 2:23-48.

13.

Brocher, S. , Artola, A. , and Singer, W. (1992). Agonists of cholinergic and noradrenergic receptors facilitate synergistically the induction of long-term potentiation in slices of rat visual cortex. Brain Research, 573(1):27-36.

14.

Crow, T.J. and Wendlandt, S. (1976). Impaired acquisition of a passive avoidance response after lesions induced in the locus coeruleus by 6-OHdopamine. Nature, 259:42-44.

15.

Damasio, A.R. (1995). Descartes' Error: Emotion, Reason, and the Human Brain. Avon Books.

16.

Edelman, G.M. (1987). Neural Darwinism Basic Books , New York.

17.

Edelman, G.M. , Reeke, Jr., G.N. , Gall, W.E. , Tononi, G. , Williams, D. , and Sporns, O. (1992). Synthetic neural modeling applied to a real-world artifact. Proceedings of the National Academy of Sciences USA, 89:7267-7271.

18.

Foote, S.L. and Morrison, J.H. (1987). Extrathalamic modulation of cortical function . Annual Review of Neuroscience, 10:76-95.

19.

Friston, K.J. , Tononi G. , Reeke, Jr., G.N. , Sporns, O. , and Edelman, G.M. (1994). Value-dependent selection in the brain: Simulation in a synthetic neural model. Neuroscience, 59:229-243.

20.

Hammer, M. (1993). An identified neuron mediates the unconditioned stimulus in associative olfactory learning in honeybees. Nature, 366:59-63.

21.

Hammer, M. and Menzel, R. (1995). Learning and memory in the honeybees. Journal of Neuroscience, 15(3):1617-1G30.

22.

Hasselmo, M.E. , Linster, C. , Patil, M. , Ma, D. , and Cekic, M. (1997). Noradrenergic suppression of synaptic transmission may influence cortical signal-to-noise ratio. Journal of Neurophysiology , 77:3326-3339.

23.

Jacobs, B.L. and Azmitia, E.C. (1992). Structure and function of the brain serotonin system. Physiological Review, 72(1):165-229.

24.

Jodo, E. , Chiang, C. , and Aston-Jones, G. (1998). Potent excitatory influence of prefrontal cortex activity on noradrenergic locus coeruleus neurons. Neuroscience , 83(1):63-79.

25.

Kaelbling, L.P. , Littman, M.P. , and Moore, A.W. (1996). Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4:237-285.

26.

Kirkwood, A. , Lee, H.-K. , and Bear, M.F. (1995). Co-regulation of long-term potentiation and experience-dependent synaptic plasticity in visual cortex by age and experience. Nature, 375:328-331.

27.

Klancnik, J.M. and Phillips, A.G. (1991). Modulation of synaptic plasticity in the dentate gyrus of the rat by electrical stimulation of the median raphe nucleus. Brain Research, 557(1-2):236-240.

28.

Klopf, A.H. (1988). A neuronal model of classical conditioning. Psychobiology, 16:85-125.

29.

Lindvall, O. and Björklund, A. (1984). General organization of cortical monoamine systems . In Descarries, L. , Reader, T. R , and Jasper, H. H. , editors, Monoamine Innervation of Cerebral Cortex , pages 9-40, New York. Alan R. Liss.

30.

Ljungberg, T. , Apicella, P. , and Schulz, W. (1992). responses of monkey dopamine neurons during learning of behavioral reactions. Journal of Neurophysiology. 67(1):145-163.

31.

Mackintosh, N.J. (1983). Conditioning and Associative Learning. Oxford university press.

32.

Mantz, J. , Milla, C. , Glowinski, J. , and Thierry, A.M. (1988). Differential effects of ascending neurons containing dopamine and noradrenaline in the control of spontaneous activity and of evoked responses in the rat prefrontal cortex. Neuroscience, 27(2):517-526.

33.

Mirenowicz, J. and Schultz, W. (1996). Preferential activation of midbrain dopamine neurons by appetitive rather than aversive stimuli. Nature, 379:449-451.

34.

Montague, P.R. , Dayan, P. , and Sejnowski, T.J. (1996). A framework formesencephalic dopamine systems based on predictive hebbian leaning. Journal of Neuroscience , 16(5):1936-1947.

35.

Reeke, Jr., G.N. , Sporns, O. , and Edelman, G.M. (1990). Synthetic neural modeling: The "Darwin" series of recognition automata. Proceedings of the IEEE, 78:1498-1530.

36.

Roerig, B. and Katz, L.C. (1997). Modulation of intrinsic circuits by serotonin 5-HT3 receptors in developing ferret visual cortex. Journal of Neuroscience. 17(21):8324-8338.

37.

Rucci, M. , Tononi, G. , and Edelman, G.M. (1997). Registration of neural maps through value-dependent learning: Modeling the alignment of auditory and visual maps in the barn owl's optic tectum. Journal of Neuroscience, 17:334-352.

38.

Saksida, L.M. , Raymond, S.M. , and Touretzky, D.S. (1997). Shaping robot behavior using principles from instrumental conditioning. Robotics and Autonomous Systems, 22(3/4):231-249.

39.

Scheier, C. and Lambrinos, D. (1996). Categorization in a real-world agent using haptic exploration and active perception. In Maes, P. , Mataric, M. , Meyer, J.-A. , Pollack, J. , and Wilson, S. W. , editors, From Animals to Animats 4, pages 65-74, Cambridge, MA. MIT Press.

40.

Schreiner, C.E. and Langner, G. (1997). Laminar fine structure of frequency organization in auditory midbrain. Nature, 388:383-386.

41.

Schultz, W. , Dayan, P. , and Montague, P.R. (1997). a neural substrate of prediction and reward. Science, 275:1593-1599.

42.

Sharkey, N.E. (1997). The new wave in robot learning. Robotics and Autonomous Systems, 22.

43.

Sporns, O. and Edelman, G.M. (1993). Solving bernstein's problem: A proposal for the development of coordinated movement by selection. Child Development , 64:960-981.

44.

Sutton, R.S. and Barto, A.G. (1981). Toward a modern theory of adaptive networks: Expectation and prediction. Psychological Review, 88(2):135-170.

45.

Tesauro, G. (1986). Simple models of classical conditioning. Biological Cybernetics, 55:187-200.

46.

Touretzky, D.S. and Saksida, L.M. (1997). Operand conditioning in skinnerbots. Adaptive Behavior, 5(3/4):219-247.

47.

Verschure, P.F.M.J. , Wray, J. , Sporns, O. , Tononi, G. , and Edelman, G.M. (1995). Multilevel analysis of a behaving real world artifact: an illustration of synthetic neural modeling. Robotics and Autonomous Systems, 16(2-4):247-265.

48.

Weng, J. , McClelland, J. , Pentland, A. , Sporns, O. , Stockman, I. , Sur, M. , And Thelen, E. (2001). Autonomous Mental Development by Robots and Animals . Science, 291: 599-600.

49.

Williams, G.V. and Goldman-Rakic, P. (1995). Modulation of memory fields by dopamine D1 receptors in prefrontal cortex. Nature, 376(6541):572-575.

50.

Wray, J. and Edelman, G.M. (1996). A model of color vision based on coritcal reentry . Cerebal Cortex, 6(5):701-716.