We present a computational model of learned avoidance behavior in a one-way avoidance experiment. Our model employs the reinforcement learning paradigm and a temporal-difference algorithm to implement both classically conditioned and instrumentally conditioned components. The role of the classically conditioned component is to develop an expectation of future benefit that is a function of the learning system's state and action. Competition among the instrumentally conditioned components determines the overt behavior generated by the learning system. Our model displays, in simulation, the reduced latency of the avoidance behavior during learning with continuing trials and the resistance to extinction of the avoidance response. These results are consistent with experimentally observed animal behavior. Our model extends the traditional two-process learning mechanism of Mowrer by explicitly defining the mechanisms of proprioceptive feedback, an internal clock, and generalization over the action space.
Anger, D. (1963). The role of temporal discriminations in the reinforcement of Sidman avoidance behavior. Journal of Experimental Analysis of Behavior, 6, 477–506.
2.
Barto, A. G. (1995). Adaptive critics and the basal ganglia. In J. C. Houk, J. L. Davis, & D. G. Beiser (Eds.), Models of information processing in the basal ganglia (pp. 215–232). Cambridge, MA: MIT Press.
3.
Barto, A. G. , Sutton, R. S., & Anderson, C. W. (1983). Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man and Cybernetics, 13(5), 834–846.
4.
Beninger, R. J. (1983). The role of dopamine in locomotor activity and learning. Brain Research, 287(2), 173–196.
5.
Beninger, R. J. (1989a). Dissociating the effects of altered dopaminergic function on performance and learning. Brain Research Bulletin, 23(4–5), 365–371.
6.
Beninger, R. J. (1989b). The role of serotonin and dopamine in learning to avoid aversive stimuli. In T. Archer & L. G. Nilsson (Eds.), Aversion avoidance, and anxiety: Perspective on aversively motivated behavior(pp. 265–282). Hillsdale, NJ: Erlbaum.
7.
Bolles, R. C. (1970). Species-specific defense reactions. In F. R. Brush (Ed.), Aversive conditioning and learning (pp. 183–233). New York: Academic Press.
8.
Bolles, R. C. (1972). Reinforcement expectancy, and learning. Psychological Review, 79, 394–409.
9.
Braitenberg, V. (1967). Is the cerebellar cortex a biological clock in the millisecond range?Progress in Brain Research, 25, 334–346.
10.
Klopf, A. H. (1988). A neuronal model of classical conditioning. Psychology, 16, 85–125.
11.
Klopf, A. H. , Morgan, J. S., & Weaver, S. E. (1993). A hierarchical network of control systems that learn: Modeling nervous system function during classical and instrumental conditioning. Adaptive Behavior, 1, 263–319.
12.
Landahl, H. D. (1964). An avoidance learning situation: A neural model. Bulletin of Mathematical Biophysics, 26, 83–89.
13.
Mazur, J. E. (1994). Learning and behavior(3rd ed.). Englewood Cliffs, NJ: Prentice Hall.
14.
Miller, S. , & Konorski, J. (1969). On a particular form of conditional reflex (B. F. Skinner, Trans.). Journal of the Experimental Analysis of Behavior, 12, 187–189. (Original Work Published 1928)
15.
Montague, P. R. (1997). Biological substrates of predictive machanisms in learning and action choice. In J. W. Donahoe & V. Packard Dorsel (Eds.), Neuralnetwork models of cognition: Biobehavioral foundations (pp. 405–421). Amsterdam: Elsevier.
16.
Moore, J. W. , Desmond, J. E., & Berthier, N. E. (1989). Adaptively timed conditioned responses and the cerebellum: A neural network approach. Biological Cybernetics, 62, 17–28.
17.
Mowrer, O. H. (1947). On the dual nature of learning: A re-interpretation of “conditioning” and “problem-solving”. Harvard Educational Review, 17, 102–148.
18.
Mowrer, O. H. (1973). Learning theory and behavior(2nd ed.). New York: Krieger.
19.
Pacut, A. , & Tych, W. (1982). Mathematical modelling of the reaction latency. Part III: A model of avoidance reaction latency and avoidance learning. Acta Neurobiological Experimentalis, 42(4–5), 397–420.
20.
Pavlov, I. P. (1927). Conditioned reflexes. London: Oxford University Press.
21.
Roitblat, H. (1987). Introduction to comparative cognition: New York: W. H. Freeman.
22.
Schiebel, M. E. , & Schiebel, M. A. (1958). Structural substrates for integrative patterns in the brain stem reticular core. In H. Jasper, L. D. Proctor, R. S. Knighton, W. S. Noshay, & R. T. Costell (Eds.), Reticular information of the brain (pp. 31–55). Boston: Little, Brown.
23.
Schiebel, M. E. , & Schiebel, M. A. (1967). Anatomical basis of attention mechanisms in vertebrate brains. In G. C. Quarton, T. Melnechuk, & F. O. Schmitt (Eds.), The neurosciences: A study program(pp. 577–602). New York: Rockefeller University Press.
24.
Schmajuk, N. A. (1994). Behavioral dynamics of escape and avoidance: A neural network approach. Paper presented at the Third International Conference on Simulation of Adaptive Behavior, Brighton, England.
25.
Schmajuk, N. A. , & Zanutto, B. S. (1997). Escape, avoidance, and imitation: A neural network approach. Adaptive Behavior, 6(1), 63–129.
26.
Seligman, M. E. P. , & Johnston, J. C. (1973). A cognitive theory of avoidance learning. In F. J. McGuigan & D. B. Lumsden (Eds.), Contemporary approaches to conditioning and learning. Washington, DC: Winston-Wiley.
27.
Sidman, M. (1966). Avoidance behavior. In W. K. Konig (Ed.), Operant behavior: Areas of research and application (pp. 448–498). New York: Appelton-Century-Crofts.
28.
Solomon, R. L. , Kamin, L. J., & Wynne, L. C. (1953). Traumatic avoidance learning: The outcomes of several extinction procedures with dogs. Journal of Abnormal and Social Psychology, 48, 291–302.
29.
Solomon, R. L. , & Wynne, L. C. (1954). Traumatic avoidance learning: The principles of anxiety conservation and partial irreversibility. Psychological Review, 61, 353–385.
30.
Sutton, M. A. , & Beninger, R. J. (1999). Psychopharmacology of conditioned reward: Evidence for a rewarding signal at D1-like dopamine receptors. Psychopharmacology, 144, 95–110.
31.
Sutton, R. S. (1988). Learning to predict by methods of temporal differences. Machine Learning, 3, 9–44.
32.
Thorndike, E. L. (1898). Animal intelligence. An experimental study of the associative process in animals. Psychological Monographs, 2(8).
33.
Tolman, E. C. (1932). Purposive behavior in animals and men. New York: Appleton-Century.
34.
Watkins, C. H. (1989). Learning from delayed rewards. Unpublished doctoral dissertation, Cambridge University, Cambridge.
35.
Zipser, D. (1986). A model of hipocampal learning during classical conditioning. Behavioral Neuroscience, 100, 764–776.
