We present a real-time neural network that integrates classical and operant processes to describe how animals learn to escape and avoid an aversive stimulus either by trial and error or by imitation. It is assumed that through classical conditioning animals build an internal model of the environment and that through operant conditioning animals select from alternative responses. The internal model of the environment provides predictions of the aversive stimulus based on environmental stimuli and the animal's own responses, and these predictions are used to train the operant conditioning block to generate responses that minimize the aversive stimulus. Computer simulations show that the model correctly describes many of the features that characterize escape and avoidance.
The network also is able to describe the imitation of a demonstrator by an observer. During the demonstration, a neural network representing the observer stores classical associations between environmental stimuli and the demonstrator's responses and aversive stimuli, and these associations serve to train the operant associations during the observer's performance. It is assumed that the demonstrator's responses evoke a representation of identical responses in the observer and that the demonstrator's unconditioned response to the aversive stimulus serves as an aversive reinforcer for the observer.
The network contributes to a general theory of adaptive behavior and is relevant to the design of autonomous systems that learn either through trial and error or through imitation.
Adams, C.D., & Dickinson, A. (1981). Instrumental responding following reinforcer devaluation . Quarterly Journal of Experimental Psychology, 33B, 109—122.
2.
Baum, M. (1966). Rapid extinction of an avoidance response following a period of response prevention in the avoidance apparatus. Psychological Reports, 18, 59-64.
3.
Baum, M. (1969). Extinction of avoidance response following response prevention: Some parametric investigations. Canadian Journal of Psychology, 23, 1-10.
4.
Baum, M. (1970). Extinction of avoidance responding through response prevention (flooding). Psychological Bulletin, 74, 276-284.
5.
Baum, M. (1976). Instrumental learning: Comparative studies. In M. P. Feldman & A. Broadhurst (Eds.), Theoretical and experimental bases of the behavior therapies. New York: Wiley.
Bolles, R.C. (1969). Avoidance and escape learning: Simultaneous acquisition of different responses. Journal of Comparative and Physiological Psychology, 68, 355-358.
Bolles, R.C., Moot, S.A., & Grossen, N.E. (1971). The extinction of shuttlebox avoidance. Learning and Motivation, 2, 324-333.
10.
Brush, E.S. (1957). Traumatic avoidance learning: The effects of conditioned stimulus length in a free responding situation. Journal of Comparative and Physiological Psychology, 50, 541-564.
11.
Campbell, B.A., & Kraeling, D. (1953). Response strength as a function of drive level and amount of drive reduction. Journal of Experimental Psychology , 45, 97-101.
12.
Cannon, W.B. (1932). The wisdom of the body. New York: Norton.
13.
Church, R.M., Brush, F.R., & Solomon, R.L. (1956). Traumatic avoidance learning: The effects of CS-US interval with a delayed-conditioning procedure in a free-responding situation . Journal of Comparative and Physiological Psychology, 49, 301—308.
14.
Colwill, R.M., & Rescorla, R.A. (1985). Postconditioning devaluation of a reinforcer affects instrumental responding. Journal of Experimental Psychology: Animal Behavior Processes, 11, 120-132.
15.
Curio, E. (1976). The ethology of predation. Berlin: Springer-Verlag.
16.
Davenport, D.G., & Olson, R.D. (1968). A reinterpretation of extinction in discriminated avoidance. Psychonomics Science, 13, 5-6.
17.
Davis, M., Schlesinger, L.S., & Sorenson, C.A. (1989). Temporal specificity of conditioning: Effects of different conditioned stimulus-unconditioned stimulus intervals on the fear-startled reflex. Journal of Experimental Psychology: Animal Behavior Processes, 15, 295-310.
18.
Dickinson, A. (1980). Contemporary animal learning theory. Cambridge, Engl.: Cambridge University Press.
19.
Dickinson, A. (1994). Instrumental conditioning. In N. J. Mackintosh (Ed.), Animal learning and cognition. San Diego: Academic.
20.
Ehrman, R.N., & Overmier, J.B. (1976). Dissimilarity of mechanisms for the evocation of escape and avoidance responding in dogs. Animal Learning and Behavior, 4,347-351.
21.
Gabriel, M., & Schmajuk, N.A. (1991). Neural substrate of avoidance learning in rabbits. In J. W Moore & M. Gabriel (Eds.), Neurocomputation and learning: Foundations of adaptive networks. Cambridge, MA: MIT.
22.
Galef, B.G. (1988). Imitation in animals: History, definition, and interpretation of data from the psychological laboratory. In T. R. Zentall & B. G. Galef (Eds.), Social learning: Psychological and biological perspectives. Hillsdale, NJ: Lawrence Erlbaum Associates.
23.
Goddard, M. (1995). Acquisition of US-no US association in Pavlovian conditioning. Learning and Motivation, 26, 264-277.
24.
Gormezano, I., & Coleman, S.R. (1973). The law of effect and CR contingent modification of the UCS. Conditional Reflex, 8, 41-56.
25.
Gormezano, I., Kehoe, E.J., & Marshall, B.S. (1983). Twenty years of classical conditioning research with the rabbit. Progress in Psychobiology and Physiological Psychology, 10, 197— 275.
26.
Grossberg, S. (1975). A neural model of attention, reinforcement, and discrimination learning. International Review of Neurobiology , 18, 263-327.
27.
Herrnstein, R.J. (1969). Method and theory in the study of avoidance. Psychological Review, 76, 49—69.
28.
Herrnstein, R.J., & Hineline, P.N. (1966). Negative reinforcement as shock-frequency reduction. Journal of the Experimental Analysis of Behavior, 9, 421-430.
29.
Heyes, C.M. (1994). Social cognition in primates. In N. J. Mackintosh (Ed.), Animal learning and cognition. San Diego: Academic.
30.
Heyes, C.M., & Dawson, G.R. (1990). A demonstration of observational learning in rats using a bidirectional control. Quarterly Journal of Experimental Psychology, 42B, 59-71.
31.
Heyes, C.M., Dawson, G.R., & Nokes, T. (1992). Imitation in rats: Initial responding and transfer evidence. Quarterly Journal of Experimental Psychology, 45B, 81-92.
32.
Hull, C.L. (1929). A functional interpretation of the conditioned reflex. Psychological Review, 36, 498-511.
33.
Hull, C.L. (1951). Essentials of behavior. Westport, CT: Greenwood.
34.
Kamin, L.J. (1954). Traumatic avoidance learning: The effects of CS-US interval with a trace-conditioning procedure. Journal of Comparative and Physiological Psychology, 47, 65-72.
35.
Kamin, L.J. (1957). The gradient of delay of secondary reward in avoidance learning. Journal of Comparative and Physiological Psychology , 50, 445-449.
36.
Kamin, L.J., Brimer, C.J., & Black, A.H. (1963). Conditioned suppression as a monitor of fear of the CS in the course of avoidance training. Journal of Comparative and Physiological Psychology, 56, 497—501.
37.
Kehoe, E.J. (1988). A layered network model of associative learning: Learning to learn and configuration. Psychological Review , 95, 411-433.
38.
Klopfer, P. (1956). An experiment on emphatic learning in ducks. American Naturalist, 91, 61-63.
39.
Konorski, J. (1967). Integrative activity of the brain. Chicago: University of Chicago.
40.
Kymissis, E., & Poulson, C.L. (1990). The history of imitation in learning theory: The language acquisition process. Journal of the Experimental Analysis of Behavior, 54, 113—127.
41.
Levis, D.J. (1989). The case for a return to a two-factor theory of avoidance: The failure of non-fear interpretations. In S. B. Klein & R. R. Mowrer (Eds.), Contemporary learning theories. Hillsdale, NJ: Lawrence Erlbaum Associates.
42.
Mackintosh, N.J. (1974). The psychology of animal learning. London: Academic.
43.
Maier, S.F., & Seligman, M.E.P. (1976). Learned helplessness: Theory and evidence. Journal of Experimental Psychology: General , 105, 3-46.
44.
Marler, P. (1970). A comparative approach to vocal learning: Song development in white-crowned sparrows. Journal of Comparative and Physiological Psychology, 71, 1-25.
45.
Marler, P., and Peters, S. (1981). Sparrows learn adult song and more from memory . Science, 213, 780-782.
46.
May, M.A. (1948). Experimentally acquired drives. Journal of Experimental Psychology, 38, 66-77.
47.
McFarland, D., & Bosser, T. (1993). Intelligent behavior in animals and robots. Cambridge, MA: MIT.
48.
Meyer, J.-A., & Wilson, S. W (Eds.) (1991). From animals to animats . Cambridge, MA: MIT.
49.
Miller, N.E. (1948). Studies of fear as an acquirable drive. Journal of Experimental Psychology, 38, 89—101.
50.
Mineka, S., & Cook, M. (1993). Mechanisms involved in the observational conditioning of fear. Journal of Experimental Psychology: General, 122, 23—38.
51.
Mineka, S., Cook, M., & Miller, S. (1984). Fear conditioned with escapable and inescapable shock: Effects of a feedback stimulus. Journal of Experimental Psychology: Animal Behavior Processes, 10, 307-323.
52.
Mineka, S., & Gino, A. (1980). Disassociation between conditioned emotional response and extended avoidance performance. Learning and Motivation , 11, 476-502.
53.
Morris, R.G.M. (1974). Pavlovian conditioned inhibition of fear during shuttlebox avoidance behavior. Learning and Motivation, 5, 424-447.
54.
Mowrer, O.H. (1947). On the dual nature of learning—a reinterpretation of conditioning and problem solving. Harvard Educational Review , 17, 102-148.
55.
Mowrer, O.H. (1960). Learning theory and symbolic process. New York: Wiley.
56.
Mowrer, O.H., & Lamoreaux, R.R. (1946). Fear as an intervening variable in avoidance conditioning. Journal of Comparative Psychology, 39, 29—50.
57.
Overmier, J.B., & Bull, J.A. (1969). On the independence of stimulus control of avoidance . Journal of Experimental Psychology, 79, 464—467.
58.
Overmier, J.B., Bull, J.A., III, and Trapold, M.A. (1971). On instrumental response interaction as explaining the influences of Pavlovian CSs upon avoidance behavior . Learning and Motivation, 2, 103-112.
59.
Overmier, J.B., & Seligman, M. E. P. (1967). Effects of inescapable shock upon subsequent escape and avoidance responding. Journal of Comparative and Physiological Psychology, 63, 28-33.
60.
Page, H.A., & Hall, J.F. (1953). Experimental extinction as a function of the prevention of response. Journal of Comparative and Physiological Psychology , 46, 33-34.
61.
Pearce, J.M., & Hall, G. (1980). A model for Pavlovian learning: Variations in the effectiveness of conditioned but not unconditioned stimuli. Psychological Review, 87, 532—552.
62.
Piaget, J. (1959). La formation du symbole chez l'enfant: Imitation, jeu et reve. Image et representation. Neuchâtel, Switzerland : Delachaux & Niestle.
63.
Rescorla, R.A. (1967). Inhibition of delay in Pavlovian fear conditioning . journal of Comparative and Physiological Psychology, 64, 114-120.
64.
Rescorla, R.A., & LoLordo, V.M. (1965). Inhibition of avoidance behavior. journal of Comparative and Physiological Psychology, 59, 406-412.
65.
Rescorla, R.A., & Wagner, A.R. (1972). A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. In A. H. Black & W F. Prokasy (Eds.), Classical conditioning II: Current research and theory. New York: Appleton-Century-Crofts .
66.
Rosellini, R.A., De Cola, J.P., & Warren, D.A. (1986). The effect of feedback stimuli on contextual fear depends upon the length of the minimum ITI. Learning and Motivation , 17, 229-242.
67.
Rudy, J.W., & Sutherland, R.J. (1989). The hippocampal formation is necessary for rats to learn and remember configural discriminations. Behavioural Brain Research, 34,97-109.
68.
Schmajuk, N.A. (1994). Behavioral dynamics of escape and avoidance: A neural network approach. In D. Cliff, P. Husbands, J.-A. Meyer , & S. W Wilson (Eds.), From animals to animats 3. Cambridge, MA: MIT.
69.
Schmajuk, N.A. (1996). The psychology of robots. Proceedings of the IEEE, 84, 1553-1561.
70.
Schmajuk, N.A. (1997). Animal learning and cognition: A neural network approach. New York: Cambridge University .
71.
Schmajuk, N.A., & DiCarlo, J.J. (1992). Stimulus configuration, classical conditioning, and the hippocampus. Psychological Review, 99, 268—305.
72.
Schmajuk, N.A., Lamoureux, J., & Holland, P.C. (in press,a). Occasion setting and stimulus configuration: A neural network approach. Psychological Review.
73.
Schmajuk, N.A., Urry, D., & Zanutto, B.S. (in press,b). The frightening complexity of avoidance: An adaptive neural network. In J. E. R. Staddon & C. Wynne (Eds.), Models of action. Hillsdale, NJ: Lawrence Erlbaum Associates.
74.
Schmajuk, N.A., Zanutto, B.S., & Buhusi, C.V. (1997). Occasion setting and operant conditioning. In P Wang (Ed.), Proceedings of the Joint Conference of Information Sciences, Vol. 2: Computational intelligence, neural networks and semiotics
75.
Schneiderman, N. (1966). Interstimulus interval function of the nictitating membrane response of the rabbbit under delay versus trace conditioning. Journal of Comparative and Physiological Psychology, 62, 397-402.
76.
Schoenfeld, W.N. (1950). An experimental approach to anxiety, escape, and avoidance behavior. In P H. Hoch and J. Zubin (Eds.), Anxiety. New York: Grune & Stratton.
77.
Seligman, M.E.P. (1975). Helplessness: On depression, development, and death. San Francisco: Freeman.
78.
Seligman, M.E.P., & Campbell, B.A. (1965). Effects of intensity and duration of punishment on extinction and avoidance response. Journal of Comparative and Physiological Psychology, 59, 295-297.
79.
Seligman, M.E.P., & Johnston, J.C. (1973). A cognitive theory of avoidance learning. In F. J. McGuigan & D. B. Lumsden (Eds.), Contemporary approaches to conditioning and learning. Washington, DC : Winston-Wiley.
80.
Seligman, M.E.P., & Maier, S.F. (1967). Failure to escape traumatic shock. journal of Experimental Psychology, 74, 1-9.
81.
Seligman, M.E.P., Rosellini, R.A., & Kozak, M.J. (1975). Learned helplessness in the rat: Time course, immunization, and reversibility. Journal of Comparative and Physiological Psychology, 88, 542-547.
82.
Sidman, M. (1953). Two temporal parameters of the maintenance of avoidance behavior by the white rat. Journal of Comparative and Physiological Psychology, 46, 253-261.
83.
Smith, M.C. (1968). CS-US interval and US intensity in classical conditioning of the rabbit's nictitating membrane response. Journal of Comparative and Physiological Psychology, 66, 679-687.
84.
Sokolov, E.N. (1960). Neuronal models and the orienting reflex. In M. A. B. Brazier (Ed.), The central nervous system and behavior. New York: Macy Foundation.
85.
Solomon, R.L., Kamin, L.J., & Wynne, L.C. (1953). Traumatic avoidance learning: The outcomes of several extinction procedures with dogs. Journal of Abnormal and Social Psychology, 48, 291-302.
86.
Solomon, R.L., & Wynne, L.C. (1953). Traumatic avoidance learning: Acquisition in normal dogs. Psychological Monographs, 67, 354.
87.
Solomon, R.L., & Wynne, L.C. (1954). Traumatic avoidance learning: The principles of anxiety conservation and partial reversibility. Psychological Review, 61, 353-385.
88.
Soltysik, S. (1985). Protection from extinction: New data and a hypothesis of several varieties of conditioned inhibition. In R. R. Miller & N. E. Spear (Eds.), Information processing in animals: Conditioned inhibition. Hillsdale, NJ: Lawrence Erlbaum Associates .
89.
Starr, M.D., & Mineka, S. (1977). Determinants of fear over the cause of avoidance learning. Learning and Motivation, 8, 332-350.
90.
Sutton, R. (1991). Reinforcement learning architectures for animats . In J.-A. Meyer & S. Wilson (Eds.), From animals to animats . Cambridge, MA: MIT.
91.
Sutton, R.S., & Barto, A.G. (1981). Toward a modern theory of adaptive networks: Expectation and prediction. Psychological Review, 88, 135-170.
92.
Thorndike, E.L. (1931). Human learning. New York: Century.
93.
Weisman, R.G., & Litner, J.S. (1969). Positive conditioned reinforcement of Sidman avoidance in rats. Journal of Comparative and Physiological Psychology , 68, 597-603.
94.
Weisman, R.G., & Litner, J.S. (1972). The role of Pavlovian events in avoidance training . In R. A. Boakes & M. S. Halliday (Eds.), Inhibition and learning . London: Academic.
95.
Widrow, B., & Hoff, M.E. (1960). Adaptive switching circuits. 1960 Institute of Radio Engineers, Western Electronic Show and Convention Record , 96-104.
96.
Young, G.A. (1975). Electrical activity of the dorsal hippocampus in rats operantly trained to lever press and to lick. Journal of Comparative and Physiological Psychology, 90, 78—90.
97.
Zhuikov, A.Y., Couvillon, P.A., & Bitterman, M.E. (1994). Quantitative two-process analysis of avoidance conditioning in goldfish. Journal of Experimental Psychology: Animal Behavior Processes, 20, 32—43.
98.
Zimmer-Hart, C.L., & Rescorla, R.A. (1974). Extinction of Pavlovian conditioned inhibition. Journal of Comparative and Physiological Psychology, 86, 837-845.
