Back to optimality: a formal framework to express the dynamics of learning optimal behavior

Abstract

Whether animals behave optimally is an open question of great importance, both theoretically and in practice. Attempts to answer this question focus on two aspects of the optimization problem, the quantity to be optimized and the optimization process itself. In this paper, we assume the abstract concept of cost as the quantity to be minimized and propose a reinforcement learning algorithm, called Value-Gradient Learning (VGL), as a computational model of behavior optimality. We prove that, unlike standard models of Reinforcement Learning, Temporal Difference in particular, VGL is guaranteed to converge to optimality under certain conditions. The core of the proof is the mathematical equivalence of VGL and Pontryagin’s Minimum Principle, a well-known optimization technique in systems and control theory. Given the similarity between VGL’s formulation and regulatory models of behavior, we argue that our algorithm may provide psychologists with a tool to formulate such models in optimization terms.

Keywords

Optimality Principle of Least Action bliss point reinforcement learning Value-Gradient Learning

Get full access to this article

View all access options for this article.

References

Allison

(1983). Behavioral economics. New York: Praeger.

Anttila

Annila

(2011). Natural games. Physics Letters, 375, 3755–3761.

Ashby

W. R.

(1952). Design for a brain: The origin of adaptive behavior. London, UK: Chapman and Hall.

Baez

J. C.

(2005). Lectures on classic mechanics. Department of Physics and Astronomy, Louisiana State University.

Barto

A. G.

Sutton

R. S.

(1982). Simulation of anticipatory responses in classical conditioning by a neuron-like adaptive element. Behavioral Brain Research, 4, 221–235.

Bellman

(1957). Dynamic programming. Princeton, NJ: Princeton University Press.

de Maupertuis

P. L. M.

(1744). Accord de différentes lois de la nature qui avaient jusqu’ici paru incompatibles. Mém. As. Sc. Paris, 417.

Domjan

(1994). Formulation of a behavior system for sexual conditioning. Psychonomic Bulletin & Review, 1, 421–428.

Euler

(1744). Methodus inveniendi Lineas curvas maximi minive proprietate gaudentes. Lausanne, Switzerland: Bousquet [reprinted in Leonhardi Euleri Opera Omnia. Series 1, vol. 24. Zurich, Switzerland: Orell Fuessli, 1952].

10.

Fairbank

(2014). Value-gradient learning. Doctoral dissertation, City University London. http://openaccess.city.ac.uk/3438/

11.

Fairbank

Alonso

(2012a). The divergence of reinforcement learning algorithms with value-iteration and function approximation. In Proceedings of the IEEE International Joint Conference on Neural Networks 2012 (IJCNN’12) (pp. 3070–3077). New York: IEEE Press.

12.

Fairbank

Alonso

(2012b).Value-gradient learning. In Proceedings of the IEEE International Joint Conference on Neural Networks 2012 (IJCNN’12) (pp. 3062–3069). New York: IEEE Press.

13.

Fairbank

Alonso

(2012c). A comparison of learning speed and ability to cope without exploration between DHP and TD(0). In Proceedings of the IEEE International Joint Conference on Neural Networks 2012 (IJCNN’12) (pp. 1478–1485). New York: IEEE Press.

14.

Fairbank

Alonso

Prokhorov

(2013). An equivalence between adaptive dynamic programming with a critic and backpropagation through time. IEEE Transactions on Neural Networks and Learning Systems, 24(12), 2088–2100.

15.

Fairbank

Prokhorov

Alonso

(2013). Approximating optimal control with value gradient learning. In Lewis

F. L.

Liu

(Eds.), Reinforcement learning and approximate dynamic programming for feedback control (pp. 142–161). New York: Wiley–IEEE Press.

16.

Fanselow

M. S.

(1994). Neural organization of the defensive behavior system responsible for fear. Psychonomic Bulletin & Review, 1, 429–438.

17.

Feynman

R. P.

Hibbs

A. R.

(1965). Quantum mechanics and path integrals. New York: McGraw-Hill.

18.

Goldstein

(1974). Classical mechanics. Cambridge, MA: Addison-Wesley.

19.

Hanson

S. J.

(1977). The Rescorla–Wagner model and the temporal control of behavior. Master’s thesis, Arizona State University.

20.

Hanson

S. J.

Timberlake

(1983). Regulation during challenge: A general model of learned performance under schedule constraint. Animal Learning & Behavior, 14, 435–442.

21.

Herrnstein

R. J.

(1961). Relative and absolute strength of responses as a function of frequency of reinforcement. Journal of the Experimental Analysis of Behaviour, 4, 267–72.

22.

Kaila

V. R. I.

Annila

(2008). Natural selection for least action. Proceedings of the Royal Society A, 464, 3055–3070.

23.

Killeen

P. R.

(1992). Mechanics of the animate. Journal of the Experimental Analysis of Behavior, 57, 429–463.

24.

Killeen

P. R.

Sitomer

M. T.

(2003). MPR. Behavioural Processes, 62, 49–64.

25.

Landau

L. D.

Lifshitz

E. M.

(1975). The classical theory of fields. London, UK: Pergamon.

26.

Ludvig

E. A.

Bellemare

M. G.

Pearson

K. G.

(2011). A primer on reinforcement learning in the brain: Psychological, computational, and neural perspectives. In Alonso

Mondragón

(Eds.), Computational neuroscience for advancing artificial intelligence: Models, methods and applications, pp. 111–144. Hershey, PA: IGI Global.

27.

Marr

Poggio

(1976). From understanding computation to understanding neural circuitry. Artificial Intelligence Laboratory. AI Memo, MIT, AIM-357.

28.

Pontryagin

L. S.

Boltyanskii

V. G.

Gamkrelidze

R. V.

Mishchenko

E. F.

(1962). The mathematical theory of optimal processes. New York: Gordon and Breach.

29.

Prokhorov

Wunsch

(1997). Adaptive critic designs. IEEE Transactions on Neural Networks, September: 997–1007.

30.

Skinner

B. F.

(1959). Cumulative record. New York: Apple-Century-Crofts.

31.

Staddon

J. E.

(1979). Operant behavior as adaptation to constraint. Journal of Experimental Psychology: General, 108, 48–67.

32.

Staddon

J. E.

(2007). Is animal behavior optimal? In Bejan

Merkx

G. W.

(Eds.), Constructal theory of social dynamics. New York: Springer.

33.

Staddon

J. E.

Cerutti

D. T.

(2003). Operant behavior. Annual Review of Psychology, 54, 115–144.

34.

Sutton

R. S.

Barto

A. G.

(1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press.

35.

Taylor

E. F.

Wheeler

J. A.

(2000). Exploring black holes: An introduction to general relativity. New York: Addison-Wesley Longman.

36.

Timberlake

(1984). Behavior regulation and learned performance: Some misapprehensions and disagreements. Journal of the Experimental Analysis of Behavior, 41, 355–375.

37.

Timberlake

Silva

K. M.

(1995). Appetitive behavior in psychology, ethology, and behavior systems. In Thompson

(Ed.), Perspectives in ethology: Vol. 11. Behavioral design (pp. 212–254). New York: Plenum.

38.

Todorov

(2006). Optimal control theory. In Doya

Ishii

Pouget

Rao

R. P. N.

(Eds.), Bayesian brain: Probabilistic approaches to neural coding (pp. 269–298). Cambridge, MA: MIT Press.

39.

Wang

F.-Y.

Zhange

Liu

(2009). Adaptive dynamic programming: An introduction. IEEE Computational Intelligence Magazine, 39–47.

40.

Watkins

C. J. C. H.

(1989). Learning from delayed rewards. Ph.D. thesis, Cambridge University.

41.

Werbos

P. J.

(1977). Advanced forecasting for global crisis warning and models of intelligence. In General Systems Yearbook.

42.

Werbos

P. J.

(1992). Approximating dynamic programming for real-time control and neural modeling. In White

Sofge

D. A.

(Eds.), Handbook of intelligent control (pp. 493–525). New York: Van Nostrand Reinhold.

43.

Wörgötter

Porr

(2005). Temporal sequence learning, prediction and control—A review of different models and their relation to biological mechanisms. Neural Computation, 17, 245–319.