Instrumental (or operant) conditioning, a form of animal learning, is similar to reinforcement learning (Watkins, 1989) in that it allows an agent to adapt its actions to gain maximally from the environment while being rewarded only for correct performance. However, animals learn much more complicated behaviors through instrumental conditioning than robots presently acquire through reinforcement learning. We describe a new computational model of the conditioning process that attempts to capture some of the aspects that are missing from simple reinforcement learning: conditioned reinforcers, shifting reinforcement contingencies, explicit action sequencing, and state space refinement. We apply our model to a task commonly used to study working memory in rats and monkeys—the delayed match-to-sample task. Animals learn this task in stages. In simulation, our model also acquires the task in stages, in a similar manner. We have used the model to train an RWI B21 robot.
Asada, M., Uchibe, E., Noda, S., Tawaratsumida, S., & Hosoda, K. (1994). Coordination of multiple behaviors acquired by a vision-based reinforcement learning. In Proceedings of the International Conference on Intelligent Robots and Systems, Munich, Germany . Los Alamitos, CA: IEEE Press .
2.
Barnett, S. (1981). Modern ethology. London: Oxford University Press.
3.
Barto, A.G., & Sutton, R.S. (1990). Time-derivative models of Pavlovian conditioning . In M. Gabriel & J. Moore (Eds.), Learning and computational neuroscience: Foundations of adaptive networks. Cambridge, MA: MIT Press.
4.
Baxter, D.A., Buonomano, D.V., Raymond, J.L., Cook, D.G., Kuenzi, F.M., Carew, T.J., & Byrne, J.H. (1991). Empirically derived adaptive elements and networks simulate associative learning. In M. L. Commons , S. Grossberg & J. E. R. Staddon (Eds.), Neural Network Models of Conditioning and Action. Hillsdale, NJ: Lawrence Erlbaum Associates.
5.
Berridge, K., Fentress, J.C., & Parr, H. (1987). Natural syntax rules control action sequence of rats. Behavioural Brain Research, 23, 59-68.
6.
Blough, D. (1959). Delayed matching in the pigeon. Journal of the Experimental Analysis of Behavior, 2, 151-160.
7.
Breland, K., & Breland, M. (1961). The misbehavior of organisms. American Psychologist, 16, 681-684.
8.
Brown, P., & Jenkins, H. (1968). Auto-shaping of the pigeon's keypeck. Journal of the Experimental Analysis of Behavior, 11, 1-8.
9.
Bussey, T.J., Muir, J.L., & Robbins, T.W. (1994). A novel automated touch-screen procedure for assessing learning in the rat using computer graphic stimuli. Neuroscience Research Communications, 15(2), 103-109.
10.
Catania, A. C., & Harnad, S. (Eds.). (1988). The selection of behavior . London: Cambridge University Press .
11.
Canine Companions for Independence (1995). The CCI program. Canine Companions for Independence, Santa Rosa, CA. Informational page available at http://grunt.berkeley.edu/cci/cci.html .
12.
Chomsky, N. (1959). Review of Skinner's verbal behavior. Language, 35, 26-58.
13.
Colombetti, M., Dorigo, M., & Borghi, G. (1996). Behavior analysis and training: A methodology for behavior engineering. IEEE Transactions on Systems, Man, and Cybernetics [B], 26(3), 365-380.
14.
Dickinson, A. (1995). Instrumental conditioning. In N. J. Mackintosh (Ed.), Handbook of perception and cognition (Vol. 9.) Orlando, FL: Academic Press.
15.
Dorigo, M., & Columbetti, M. (1994). Robot shaping: Developing autonomous agents through learning. Artificial Intelligence , 70(2), 321-370.
16.
Drescher, G.L. (1991). Made-up minds. Cambridge, MA: MIT Press.
17.
Gallistel, C.R. (1990). The organization of learning. Cambridge, MA: MIT Press.
18.
Gollub, L. (1977). Conditioned reinforcement: Schedule effects. In W. Honig & J. Staddon (Eds.), Handbook of operant behavior. Mt. Kisco, NY: Prentice Hall.
19.
Graham, J., Alloway, T., and Krames, L. (1994). Sniffy, the virtual rat: Simulated operant conditioning . Behavior Research Methods, Instruments, & Computers , 26(2), 134-141.
20.
Grossberg, S. (1972). A neural theory of punishment and avoidance. II. Quantitative theory. Mathematical Biosciences, 15, 253-285.
21.
Gutnikov, S., Barnes, J., & Rawlins, J. (1994). Working memory tasks in five-choice operant chambers: Use of relative and absolute spatial memories. Behavioral Neuroscience, 108(5), 899-910.
22.
Hammer, M. (1993). An identified neuron mediates the unconditioned stimulus in associative olfactory learning in honeybees. Nature, 366, 59-63.
23.
Hampson, R.E., Heyser, C.J., & Deadwyler, S.A. (1993). Hippocampal cell firing correlates of delayed-match-to-sample performance in the rat. Behavioral Neuroscience, 107(5), 715-739.
24.
Holland, P.C. (1993). Cognitive aspects of classical conditioning. Current Opinion in Neurobiology, 3, 230-236.
25.
Klopf, A.H. (1988). A neuronal model of classical conditioning. Psychobiology, 16, 85-125.
26.
Krames, L., Graham, J., & Alloway, T. (1995). Sniffy, the virtual rat. Pacific Grove, CA: Brooks/Cole. [Includes software diskette.]
27.
Lashley, K. (1951). The problem of serial order in behavior. In L. Jeffries (Ed.), Cerebral mechanisms in behavior. New York: Wiley.
28.
Mahadevan, S., & Connell, J. (1992). Automatic programming of behavior-based robots using reinforcement learning. Artificial Intelligence, 55, 311-365.
29.
Maki, W.S., & Abunawass, A.M. (1991). A connectionist approach to conditional discriminations: Learning, short-term memory, and attention . In M. L. Commons, S. Grossberg & J. E. R. Staddon (Eds.), Neural Network Models of Conditioning and Action. Hillsdale, NJ: Lawrence Erlbaum Associates.
30.
Matarić, M.J. (1995). Designing and understanding adaptive group behavior . Adaptive Behavior, 4(1), 50-81.
Millán, J. d. R. (1994). Learning efficient reactive behavioral sequences from basic reflexes in a goal-directed autonomous robot. In D. Cliff, P. Husbands, J.-A. Meyer & S. W Wilson (Eds.), From animals to animats 3: Proceedings of the Third International Conference, on Simulation of Adaptive Behavior. Cambridge, MA: MIT Press.
33.
Millán, J. d. R. (1996). Rapid, safe, and incremental learning of navigation strategies. IEEE Transactions on Systems, Man, and Cybernetics [B], 26(3), 408-420.
34.
Montague, R.R., Dayan, P., Person, C., & Sejnowski, T.J. (1995). Bee foraging in uncertain environments using predictive Hebbian learning. Nature, 377, 725-728. Pavlov, I.P. (1927). Conditioned reflexes . London: Oxford University Press .
35.
Pryor, K. (1975). Lads before the wind. New York : Harper & Row.
36.
Raymond, J.L., Baxter, D.A., Buonomano, D.V., & Byrne, J.H. (1992). A learning rule based on empirically derived activity-dependent neuromodulation supports operant conditioning in a small network. Neural Networks, 5(5), 789-803.
37.
Rescorla, R.A., & Wagner, A.R. (1972). A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. In A. H. Black & W F Prokasy (Eds.), Classical conditioning II: Theory and research . New York: Appleton-Century-Crofts .
38.
Reynolds, G.S. (1968). A primer of operant conditioning. Glenview, IL: Scott, Fores-man.
39.
Schmajuk, N.A., & Urry, D.W. (1995). The frightening complexity of avoidance: An adaptive neural network. In C. Wynne & J. Staddon (Eds.), Models of action: Mechanisms for adaptive behavior. Hillsdale, NJ: Lawrence Erlbaum Associates.
40.
Schultz, W., Romo, R., Ljungberg, T., Mirenowicz, J., Hollerman, J.R., & Dickinson, A. (1995). Reward-related signals carried by dopamine neurons . In J. C. Houk, J. L. Davis, & D. G. Beiser (Eds.), Models of information processing in the basal ganglia. Cambridge, MA: MIT Press.
41.
Schwartz, B. (1989). Psychology of learning and behavior. New York: Norton.
42.
Simmons, R. (1994). Structured control for autonomous robots. IEEE Transactions on Robotics and Automation, 10(1), 34-43.
43.
Singh, S. (1992). Transfer of learning across sequential tasks. Machine Learning, 8, 323-339.
44.
Skinner, B. (1938). Behavior of organisms. New York: Appleton-Century-Crofts.
45.
Skinner, B. (1948). "Superstition" in the pigeon. Journal of Experimental Psychology, 38, 168-172.
46.
Staddon, J., & Simmelhag, V. (1971). The "superstition" experiment: A reexamination of its implications for the principle of adaptive behavior. Psychological Review, 78, 3-43.
47.
Sutton, R.S., & Barto, A.G. (1981). Toward a modern theory of adaptive networks: Expectation and prediction. Psychological Review, 88, 135-170.
48.
Touretzky, D.S., & Saksida, L. (1996). Skinnerbots. In P Maes, M. Matarić, J.-A. Meyer, J. Pollack, & S. W Wilson (Eds.), From animals to animats 4: Proceedings of the Fourth International Conference on Simulation of Adaptive Behavior. Cambridge, MA: MIT Press.
49.
Verschure, P F. M. J., Wray, J., Sporns, O., Tononi, G., & Edelman, G.M. (1995). Multilevel analysis of classical conditioning in a real world artifact. Robotics and Autonomous Systems , 16(2-4), 247-265.
