A computational model of nervous system function during classical and instrumental conditioning is proposed. The model assumes the form of a hierarchical network of control systems. Each control system is capable of learning and is referred to as an associative control process (ACP). Learning systems consisting of ACP networks, employing the drive-reinforcement learning mechanism (Klopf, 1988) and engaging in real-time, closed-loop, goal-seeking interactions with environments, are capable of being classically and instrumentally conditioned, as demonstrated by means of computer simulations. In multiple-T mazes, the systems learn to chain responses that avoid punishment and that lead eventually to reward. The temporal order in which the responses are learned and extinguished during instrumental conditioning is consistent with that observed in animal learning. Also consistent with animal learning experimental evidence, the ACP network model accounts for a wide range of classical conditioning phenomena. ACP networks, at their current stage of development, are intended to model sensorimotor, limbic, and hypothalamic nervous system function, suggesting a relationship between classical and instrumental conditioning that extends Mowrer's (1956, 1960a/1973) two-factor theory of learning. In conjunction with consideration of limbic system and hypothalamic function, the role of emotion in natural intelligence is modeled and discussed. ACP networks constitute solutions to temporal and structural credit assignment problems, suggesting a theoretical approach for the synthesis of machine intelligence.
Arbib, M. (1972). The metaphorical brain. New York: Wiley.
2.
Ashby, W.R. (1952). Design for a brain. New York : Wiley.
3.
Baird, L.C., & Baker, W.L. (1990). A connectionist system for nonlinear control. Proceedings of the American Institute of Aeronautics and Astronautics Conference on Guidance, Navigation, and Control. Boston, MA.
4.
Baker, W.L., & Farrell, J.A. (1992). An introduction to connectionist learning control systems. In D. A. White & D. A. Sofge (Eds.), The handbook of intelligent control. New York: Van Nostrand Reinhold.
5.
Bartha, G.T., Thompson, R.F., & Gluck, M.A. (1991). Sensorimotor learning and the cerebellum. In M. Arbib & J. Ewert (Eds.), Visual structures and integrated functions . Berlin: Springer-Verlag.
6.
Barto, A.G. (1991). Some learning tasks from a control perspective . In L. Nadel & D. Stein (Eds.), 1990 Lectures in complex systems. Redwood City, CA: Addison-Wesley.
7.
Barto, A.G., & Sutton, R.S. (1981). Goal seeking components for adaptive intelligence: An initial assessment (Technical Report Number 81-1070). Wright-Patterson Air Force Base, OH: Air Force Wright Aeronautical Laboratories. (DTIC Report AD 101476 available from the Defense Technical Information Center, Cameron Station, Alexandria, VA 22304-6145.)
8.
Barto, A.G., Sutton, R.S., & Anderson, C.W. (1983). Neuronlike elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics, 13, 835-846.
9.
[Reprinted in Anderson, J. A., & Rosenfeld, E. (Eds.). (1988), Neurocomputing: Foundations of research. Cambridge, MA: MIT Press.]
10.
Barto, A.G., Sutton, R.S., & Watkins, C.J.C.H. (1990). Learning and sequential decision making. In M. Gabriel & J. Moore (Eds.), Learning and computational neuroscience: Foundations of adaptive networks. Cambridge, MA: MIT Press.
11.
Beer, R.D. (1990). Intelligence as adaptive behavior: An experiment in computational neuroethology. San Diego: Academic Press.
12.
Berridge, K.C., & Grill, H.J. (1983). Alternating ingestive and aversive consummatory responses suggest a two-dimensional analysis of palatability in rats. Behavioral Neuroscience, 97(4), 563-573.
13.
Bindra, D. (1972). A unified account of classical conditioning and operant training. In A. H. Black & W F Prokasy (Eds.), Classical conditioning II: Current research and theory. New York : Appleton-Century-Crofts.
14.
Booker, L.B. (1982). Intelligent behavior as an adaptation to the task environment (Doctoral dissertation, University of Michigan , 1982). Dissertation Abstracts International, 43, 469B.
Carew, T.J., & Sahley, C.L. (1986). Invertebrate learning and memory: From behavior to molecules. Annual Review of Neuroscience, 9, 435-487.
18.
Carlson, N.R. (1986). Physiology of behavior (3rd ed.). Boston: Allyn and Bacon .
19.
Carpenter, W.B. (1874). Principles of mental physiology, with their applications to the training and discipline of the mind and the study of its morbid conditions . New York: Appleton-Century-Crofts .
20.
Cliff, D.T. (1990). Computational neuroethology: A provisional manifesto (Cognitive Science Research Paper 162). Brighton, England: School of Cognitive and Computing Sciences, The University of Sussex.
21.
Colwill, R.M., & Rescorla, R.A. (1990). Evidence for the hierarchical structure of instrumental learning. Animal Learning and Behavior , 18(1), 71-82.
22.
Cooper, J.R., Bloom, F.E., & Roth, R.H. (1991). The biochemical basis of neuropharmacology. New York: Oxford University Press.
23.
Desmond, J.E. (1990). Temporally adaptive responses in neural models: The stimulus trace. In M. Gabriel & J. Moore (Eds.), Learning and computational neuroscience: Foundations of adaptive networks. Cambridge, MA: MIT Press.
24.
Dickinson, A. (1980). Contemporary animal learning theory. Cambridge: Cambridge University Press.
25.
Dreyfus, H.L. (1992). What computers still can't do: A critique of artificial reason. Cambridge, MA: MIT Press .
26.
Flaherty, C.F. (1985). Animal learning and cognition. New York: Knopf.
27.
Fonberg, E. (1986). Amygdala, emotions, motivation, and depressive states. In R. Plutchik & H. Kellerman (Eds.), Emotion: Theory, research, and experience, Vol. 3. New York: Academic Press.
28.
Ford, E.E. (1989). Fostering self-control: Comments of a counselor . In W A. Hershberger (Ed.), Volitional action. New York: Elsevier Science Publishers.
29.
Frese, M., & Sabini, J. (Eds.). (1985). Goal directed behavior: The concept of action in psychology. Hillsdale, NJ: Lawrence Erlbaum.
30.
Gallistel, C.R. (1980). The organization of action: A new synthesis. Hillsdale, NJ: Lawrence Erlbaum.
31.
Gibson, J.J. (1950). The perception of the visual world. Boston: Houghton Mifflin.
32.
Gibson, J.J. (1966). The senses considered as perceptual systems. New York: Houghton Mifflin.
33.
Gibson, J.J. (1979). The ecological approach to visual perception. Boston: Houghton Mifflin.
34.
Glickman, S.E., & Schiff, B.B. (1967). A biological theory of reinforcement. Psychological Review, 74(2), 81-109.
35.
Gluck, M.A., Parker, D.B., & Reifsnider, E. (1988). Some biological implications of a differential-Hebbian learning rule. Psychobiology, 16(3), 298-302.
36.
Gray, J.A. (1975). Elements of a two-process theory of learning. New York: Academic Press.
37.
Gray, J.A. (1982). The neurophysiology of anxiety: An enquiry into the functions of the septo-hippocampal system. Oxford: Clarendon Press.
38.
Gray, J.A. (1987). The psychology of fear and stress (2nd ed.). Cambridge, Engl.: Cambridge University Press.
39.
Grossberg, S. (1971). Pavlovian pattern learning by nonlinear neural networks. Proceedings of the National Academy of Sciences , 68, 828-831.
40.
Grossberg, S. (1982). Studies of mind and brain. Boston: Reidel.
41.
Grossberg, S. (Ed.). (1987). The adaptive brain (Vols. 1 & 2). New York: North-Holland .
42.
Hall, J.F. (1989). Learning and memory (2nd ed.). Boston: Allyn and Bacon .
43.
Hampson, S.E. (1990). Connectionistic problem solving: Computational aspects of biological learning. Boston: Birkhauser.
44.
Heath, R.G. (1986). The neural substrate for emotion. In R. Plutchik & H. Kellerman (Eds.), Emotion: Theory, research, and experience, Vol. 3, New York: Academic Press.
45.
Hill, J.H. (1939). Goal gradient, anticipation, and perseveration in compound trial-and-error learning. Journal of Experimental Psychology, 25, 566-585.
46.
Hofsten, C. von (1990). A perception-action perspective on the development of manual movements. In M. Jeannerod (Ed.), Attention and performance XIII. Hillsdale, NJ: Lawrence Erlbaum.
47.
Hull, C.L. (1943). Principles of behavior. New York: Appleton-Century-Crofts.
48.
Hull, C.L. (1952). A behavior system: An introduction to behavior theory concerning the individual organism. New Haven: Yale University Press.
49.
Ito, M. ( 1989). Long-term depression. Annual Review of Neuroscience, 12, 85-102.
50.
Johnson, J.D. (1991). The selectively attentive environmental learning system. Unpublished doctoral dissertation, University of Cincinnati.
51.
Kelso, J.A.S., Delcolle, J.D., & Schoner, G. (1990). Action-perception as a pattern formation process . In M. Jeannerod (Ed.), Attention and performance XIII. Hillsdale, NJ: Lawrence Erlbaum.
52.
Klapp, S.T., Porter-Graham, K.A., & Hoifjeld, A.R. (1991). The relation of perception and motor action: Ideomotor compatibility and interference in divided attention . Journal of Motor Behavior, 23(3), 155-162.
53.
Klopf, A.H. (1972). Brain function and adaptive systems—A heterostatic theory (Report No. 133 [AFCRL-72-0164]). L. G. Hanscom Field, Bedford, MA: Air Force Cambridge Research Laboratories . (DTIC Report AD 742259, available from the Defense Technical Information Center, Cameron Station, Alexandria, VA 22304-6145.)
54.
Klopf, A.H. (1982). The hedonistic neuron: A theory of memory, learning, and intelligence. New York: Hemisphere/Taylor and Francis.
55.
Klopf, A.H. (1986). A drive-reinforcement model of single neuron function: An alternative to the Hebbian neuronal model. In J. S. Denker (Ed.), AIP Conference Proceedings 151: Neural networks for computing. New York: American Institute of Physics.
56.
Klopf, A.H. (1988). A neuronal model of classical conditioning. Psychobiology, 16(2), 85-125.
57.
Klopf, A.H. (1989). Classical conditioning phenomena predicted by a drive-reinforcement model of neuronal function. In J. H. Byrne & W O. Berry (Eds.), Neural models of plasticity: Experimental and theoretical approaches . New York: Academic Press.
58.
Klopf, A.H., & Morgan, J.S. (1990). The role of time in natural intelligence: Implications of classical and instrumental conditioning for neuronal and neural network modeling. In M. Gabriel & J. Moore (Eds.), Learning and computational neuroscience: Foundations of adaptive networks. Cambridge, MA: MIT Press.
59.
Kosko, B. (1986). Differential Hebbian learning. In J. S. Denker (Ed.), AIP Conference Proceedings 151: Neural networks for computing. New York : American Institute of Physics.
60.
Leiner, H.C., Leiner, A.L., & Dow, R.S. (1986). Does the cerebellum contribute to mental skills ? Behavioral Neuroscience, 100(4), 443-454.
61.
Levine, D.S. (1991). Introduction to neural and cognitive modeling . Hillsdale, NJ: Lawrence Erlbaum .
62.
Levitan, I.B. (1987). Protein phosphorylation and neuronal modulation . In G. Adelman (Ed.), Encyclopedia of neuroscience, Vol. 2. Boston: Birkhauser.
63.
MacKay, D.G. (1987). The organization of perception and action. New York: Springer-Verlag.
64.
Mackintosh, N.J. (1974). The psychology of animal learning. New York: Academic Press.
65.
Mackintosh, N.J. (1983). Conditioning and associative learning. Oxford: Clarendon Press.
66.
Meyer, J., & Wilson, S. W (Eds.) (1991). Proceedings of the International Workshop on the Simulation of Adaptive Behavior: From animals to animats. Cambridge, MA: MIT Press.
67.
Michaels, C.F., & Carello, C. (1981). Direct perception. Englewood Cliffs, NJ:Prentice-Hall.
68.
Miller, N.E. (1951). Learnable drives and rewards. In S. S. Stevens (Ed.), Handbook of experimental psychology. New York: Wiley.
69.
Miller, N.E. (1963). Some reflections on the law of effect produce a new alternative to drive reduction. In M. R. Jones (Ed.), Nebraska Symposium on Motivation . Lincoln: Nebraska University Press .
70.
Minsky, M.L. (1961). Steps toward artificial intelligence. Proceedings of the IRE, 49, 8-30.
71.
Moore, J.W., Berthier, N.E., & Blazis, D.E.J. (1990). Classical eye-blink conditioning : Brain systems and implementation of a computational model. In M. Gabriel & J. Moore (Eds.), Learning and computational neuroscience: Foundations of adaptive networks. Cambridge, MA: MIT Press.
72.
Moore, J.W., Desmond, J.E., Berthier, N.E., Blazis, D.E.J., Sutton, R.S., & Barto, A.G. (1986). Simulation of the classically conditioned nictitating membrane response by a neuron-like adaptive element: Response topography, neuronal firing, and interstimulus intervals. Behavioral Brain Research, 21, 143-154.
73.
Morgan, J.S., Patterson, E.C., & Klopf, A.H. (1990a). A drive-reinforcement model of simple instrumental conditioning. Proceedings of the International Joint Conference on Neural Networks, Vol. 2, 227-232.
74.
Morgan, J.S., Patterson, E.C., & Klopf, A.H. (1990b). Drive-reinforcement learning : A self-supervised model for adaptive control. Network: Computation in Neural Systems, 1, 439-448.
75.
Mowrer, O.H. (1956). Two-factor learning theory reconsidered, with special reference to secondary reinforcement and the concept of habit. Psychological Review, 63, 114-128.
76.
Mowrer, O.H. (1960a). Learning theory and behavior. New York: Wiley. (Krieger edition, 1973.)
77.
Mowrer, O.H. (1960b). Learning theory and the symbolic processes. New York: Wiley.
78.
Napier, R.M., Macrae, M., & Kehoe, E.J. (1992). Rapid reacquisition in conditioning of the rabbit's nictitating membrane response. Journal of Experimental Psychology: Animal Behavior Processes, 18(2), 182-192.
Powers, W.T. (1978). Quantitative analysis of purposive systems: Some spadework at the foundations of scientific psychology. Psychological Review, 85, 417-435.
84.
Powers, W.T., Clark, R.K., & McFarland, R.L. (1960a). A feedback model of human behavior: Part 1. Perceptual and Motor Skills, 11, 71-78.
85.
Powers, W.T., Clark, R.K., & McFarland, R.L. (1960b). A feedback model of human behavior: Part 2. Perceptual and Motor Skills, 11, 309-323.
86.
Prinz, W. (1987). Ideo-motor action. In H. Heuer & A. F. Sanders (Eds.), Perspectives on perception and action. Hillsdale, NJ: Lawrence Erlbaum.
87.
Rescorla, R.A. (1977). Pavlovian second-order conditioning: Some implications for instrumental behavior. In H. Davis & H. M. B. Hurwitz (Eds.), Operant Pavlovian interactions. Hillsdale, NJ: Lawrence Erlbaum.
88.
Rescorla, R.A. (1987). A Pavlovian analysis of goal-directed behavior . American Psychologist, 42, 119-129.
89.
Rescorla, R.A., & Solomon, R.L. (1967). Two process learning theory: Relationships between Pavlovian conditioning and instrumental learning. Psychological Review, 74(3), 151-182.
90.
Roitblat, H.L. (1987). Introduction to comparative cognition. New York: W L. Freeman.
91.
Roitblat, H.L. (1988). A cognitive action theory of learning. In J. Delacour & J. C. S. Levy (Eds.), Systems with learning and memory abilities . New York: Elsevier.
92.
Rolls, E.T. (1986). A theory of emotion, and its application to understanding the neural basis of emotion. In Y Oomura (Ed.), Emotions: Neuronal and chemical control. New York: Karger.
93.
Rosenbaum, D.A. (1987). Hierarchical organization of motor programs. In S. P. Wise (Ed.), Higher brain functions: Recent explorations of the brain's emergent properties. New York: Wiley.
94.
Schmajuk, N.A., & Urry, D.W. (1992). Avoidance revisited: A neural network approach . Proceedings of the 43rd Annual Meeting of the Psychonomic Society. St. Louis, MO.
95.
Schneirla, T.C. (1959). An evolutionary and developmental theory of biphasic processes underlying approach and withdrawal. In M. R. Jones (Ed.), Nebraska Symposium on Motivation . Lincoln, NE: University of Nebraska Press.
96.
Spence, K.W. (1960). Behavior theory and learning: Selected papers . Englewood Cliffs, NJ:Prentice-Hall .
97.
Sperry, R.W. (1952). Neurology and the mind-brain problem. American Scientist, 40, 291-312.
98.
Stein, L. (1964). Reciprocal action of reward and punishment mechanisms . In R. G. Heath (Ed.), The role of pleasure in behavior. New York: Harper & Row.
99.
Stein, L., & Belluzzi, J.D. (1988). Operant conditioning of individual neurons. In M. L. Commons, R. M. Church, J. R. Stellar , & A. R. Wagner (Eds.), Quantitative analyses of behavior, Vol. 7. Hillsdale, NJ: Lawrence Erlbaum.
100.
Sutton, R.S. (1984). Temporal credit assignment in reinforcement learning . Unpublished doctoral dissertation, University of Massachusetts , Amherst.
101.
Sutton, R.S. (1988). Learning to predict by the methods of temporal differences. Machine Learning, 3, 9-44.
102.
Sutton, R.S. (1991). Reinforcement learning architectures for animats . Proceedings of the International Workshop on the Simulation of Adaptive Behavior: From animals to animats. Cambridge, MA: MIT Press.
103.
Sutton, R.S., & Barto, A.G. (1981). Toward a modern theory of adaptive networks: Expectation and prediction. Psychological Review, 88, 135-170.
104.
Sutton, R.S., & Barto, A.G. (1987). A temporal-difference model of classical conditioning . Proceedings of the Ninth Annual Conference of the Cognitive Science Society. Hillsdale, NJ: Lawrence Erlbaum.
105.
Sutton, R.S., & Barto, A.G. (1990). Time-derivative models of Pavlovian reinforcement . In M. Gabriel & J. Moore (Eds.), Learning and computational neuroscience: Foundations of adaptive networks. Cambridge, MA: MIT Press.
106.
Sutton, R.S., Barto, A.G., & Williams, R.J. (1991). Reinforcement learning is direct adaptive optimal control. Proceedings of the 1991 American Control Conference, Vol. 3. Boston, MA.
107.
Tesauro, G. (1986). Simple neural models of classical conditioning . Biological Cybernetics, 55, 187-200.
108.
Timberlake, W., & Lucas, G.A. (1989). Behavior systems and learning: From misbehavior to general principles. In S. B. Klien & R. R. Mowrer (Eds.), Contemporary learning theories: Instrumental conditioning theory and the impact of biological constraints on learning. Hillsdale, NJ: Lawrence Erlbaum.
109.
Turvey, M.T. (1977). Preliminaries to a theory of action with reference to vision. In R. E. Shaw & J. Bransford (Eds.), Perceiving, acting, and knowing. Hillsdale, NJ: Lawrence Erlbaum.
110.
Vaccarino, F.J., Schiff, B.B., & Glickman, S.E. (1989). Biological view of reinforcement . In S. B. Klien & R. R. Mowrer (Eds.), Contemporary learning theories: Instrumental conditioning theory and the impact of biological constraints on learning. Hillsdale, NJ: Lawrence Erlbaum.
111.
Washburn, S.L., & Harding, R.S. (1970). Evolution of primate behavior. In F O. Schmitt (Ed.), The neurosciences: Second study program. New York: Rockefeller University Press.
Weil, J.L. (1974). A neurophysiological model of emotional and intentional behavior. Springfield, IL: Thomas .
114.
Werbos, P.J. (1987). Building and understanding adaptive systems: A statistical/numerical approach to factory automation and brain research. IEEE Transactions on Systems, Man, and Cybernetics, SMC-17(1), 7-20.
115.
Werbos, P.J. (1990). Consistency of HDP applied to a simple reinforcement learning problem. Neural Networks, 3, 179-189.
116.
Werbos, P.J. (1992). The cytoskeleton: Why it may be crucial to human learning and to neurocontrol. Nanobiology, 1, 75-95.
117.
Wiener, N. (1948/1961). Cybernetics or control and communication in the animal and the machine. Cambridge, MA: MIT Press.
118.
Williams, R.J., & Baird, L.C. (1990). A mathematical analysis of actor-critic architectures for learning optimal control through incremental dynamic programming. Proceedings of the Sixth Yale Workshop on Adaptive and Learning Systems. New Haven, CT: Yale University .
119.
Wilson, S.W. (1986). Knowledge growth in an artificial animal. In K. S. Narendra (Ed.), Adaptive and learning systems. New York: Plenum.
120.
Wise, R.A. (1989). The brain and reward. In J. M. Liebman & S. J. Cooper (Eds.), The neuropharmacological basis of reward. Oxford: Clarendon Press.
121.
Wordworth, R.S. (1918). Dynamic psychology. New York : Columbia University Press.
122.
Young, P.T. (1955). The role of hedonic processes in motivation. In M. R. Jones (Ed.), Nebraska Symposium on Motivation. Lincoln: University of Nebraska Press.
123.
Zagon, I.S., McLaughlin, P.J., & Smith, S. (1977). Neural populations in the cerebellum: Estimations from isolated cell nuclei. Brain Research, 127, 279-282.
