In machine learning, learning a task is expensive (many training samples are needed) and it is therefore of general interest to be able to reuse knowledge across tasks. This is the case in aerial robotics applications, where an autonomous aerial robot cannot interact with the environment hazard free. Prototype generation is a well known technique commonly used in supervised learning to help reduce the number of samples needed to learn a task. However, little is known about how such techniques can be used in a reinforcement learning task. In this work we propose an algorithm that, in order to learn a new (target) task, first generates new samples—prototypes—based on samples acquired previously in a known (source) task. The proposed approach uses Gaussian processes to learn a continuous multidimensional transition function, rendering the method capable of reasoning directly in continuous (states and actions) domains. We base the prototype generation on a careful selection of a subset of samples from the source task (based on known filtering techniques) and transforming such samples using the (little) knowledge acquired in the target task. Our experimental evidence gathered in known reinforcement learning benchmark tasks, as well as a challenging quadcopter to helicopter transfer task, suggests that prototype generation is feasible and, furthermore, that the filtering technique used is not as important as a correct transformation model.
AmmarH.TuylsK.TaylorM. (2012). Reinforcement learning transfer via sparse coding. In Proceedings of the 11th international conference on autonomous agents and multiagent systems (Vol. 1, pp. 383–390).
2.
AtkesonC.SantamariaJ. (1997). A comparison of direct and model-based reinforcement learning. In IEEE international conference on robotics and automation (Vol. 4, pp. 3557–3564). Piscataway, NJ: IEEE.
Bou AmmarH.EatonE.RuvoloP.TaylorM. E. (2015). Unsupervised cross-domain transfer in policy gradient reinforcement learning via manifold alignment. In Proceedings of the twenty-ninth AAAI conference on artificial intelligence (pp. 2504–2510). AAAI Press. Austin, Texas, USA. AI Access Foundation.
5.
DeisenrothM.PetersJ.RasmussenC. (2008). Approximate dynamic programming with Gaussian processes. In American control conference (pp. 4480–4485).
6.
DeisenrothM.RasmussenC. (2011). PILCO: A model-based and data-efficient approach to policy search. In GetoorL.SchefferT. (Eds.), ICML (pp. 465–472).
7.
DeisenrothM.RasmussenC.FoxD. (2011). Learning to control a low-cost manipulator using dataefficient reinforcement learning. In Proceedings of robotics: Science and systems. Los Angeles, CA.
8.
DeisenrothM.RasmussenC.PetersJ. (2008). Model-based reinforcement learning with continuous states and actions. In 16th european symposium on artificial neural networks (pp. 19–24), Bruges, Belgium.
9.
DietterichT. (2000). Hierarchical reinforcement learning with the maxq value function decomposition. Journal of Artificial Intelligence Research, 13, 227–303.
10.
DrummondC. (2002). Accelerating Reinforcement Learning by Composing Solutions of Automatically Identified Subtasks. Journal of Artificial Intelligence Research, 16, 59–104.
11.
EngelY.MannorS.MeirR. (2003). Bayes meets bellman: The gaussian process approach to temporal difference learning. In FawcettT.MishraN. (Eds.), ICML (pp. 154–161). AAAI Press, Washington D.C.
12.
EngelY.MannorS.MeirR. (2005). Reinforcement learning with Gaussian processes. In Proceedings of the 22nd international conference on Machine learning - ICML ’05 (pp. 201–208), ACM International conference proceedings series, Bonn, Germany.
13.
FergusonK.MahadevanS. (2006). Proto-transfer learning in markov decision processes using spectral methods. In Proceedings of the ICML-06 workshop on structural knowledge transfer for machine learning. Citeseer.
14.
GarciaE.Munoz de CoteE.MoralesE. (2013). Qualitative Transfer for Reinforcement Learning with Continuous State and Action Spaces. In Ruiz-SchucloperJ.Sanniti di BajaG. (Eds.), 18th. iberoamerican congress on pattern recognition (pp. 198–205). Springer-Verlag Santiago, Chile.
15.
GarciaE.Munoz de CoteE.MoralesE. (2014). Transfer learning for continuous state and action spaces. International Journal of Pattern Recognition and Artificial Intelligence, 28 1460007 (20 pages).
16.
HasseltH. (2011). Reinforcement Learning in Continuous State and Action Spaces. In Reinforcement learning: State of the art. Springer.
17.
JaakolaT.JordanM.SinghS. (1995). On the convergence of stochastic iterative dynamic programming algorithms. Neural Computation, 6, 1185–1201.
18.
LazaricA. (2008). Knowledge transfer in reinforcement learning (PhD Thesis). Politecnico di Milano, Italy.
19.
LazaricA.RestelliM.BonariniA. (2007). Reinforcement learning in continuous action spaces through sequential monte carlo methods. In Advances in neural information processing systems. Citeseer.
20.
LazaricA.RestelliM.BonariniA. (2008). Transfer of samples in batch reinforcement learning. In Proceedings of the 25th international conference on Machine learning - ICML ’08 (pp. 544–551).
21.
MartínH. J.de LopeJ.MaravallD. (2011, March). Robust high performance reinforcement learning through weighted k-nearest neighbors. Neurocomputing, 74, 1251–1259.
22.
MehtaN.NatarajanS.TadepalliP.FernA. (2008). Transfer in variable-reward hierarchical reinforcement learning. Machine Learning, 73, 289–312.
23.
Murray-SmithR.SbarbaroD. (2002). Nonlinear adaptive control using non-parametric Gaussian process prior models. In In 15th IFAC world congress on automatic control (pp. 21–26).
24.
NgA.HaradaD.RussellS. (1999). Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the sixteenth international conference on machine learning (pp. 278–287). Morgan Kaufmann.
25.
NgA.KimH.JordanM.SastryS. (2004). Autonomous helicopter flight via reinforcement learning. In ThrunS.LawrenceS. & B. S. (Eds.), Advances in neural information processing systems16. Cambridge, MA: MIT Press.
26.
PutermanM. (1994). Markov decision processes—discrete stochastic dynamic programming. New York, NY: John Wiley & Sons, Inc.
27.
RasmussenC.DeisenrothM. (2008). Probabilistic inference for fast learning in control. Recent Advances in Reinforcement Learning, 5323(November), 229–242.
28.
RasmussenC.KussM. (2004). Gaussian Processes in Reinforcement Learning. Advances in Neural Information Processing Systems, 16, pp. 751–759.
29.
RasmussenC.WilliamsC. (2006). Gaussian Processes for Machine Learning. International Journal of Neural Systems, 14(2), 69–106.
SilverD.. (Eds.). (2005). Inductive transfer: 10 years later workshop.
32.
SoniV.SinghS. (2006). Using homomorphisms to transfer options across continuous reinforcement learning domains. In AAAI (pp. 494–499).
33.
SuttonR.BartoA. (1998). Introduction to Reinforcement Learning. MIT Press.
34.
TaylorM.JongN.StoneP. (2008). Transferring instances for model-based reinforcement learning. Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer Berlin Heidelberg.
35.
TaylorM.StoneP. (2009). Transfer learning for reinforcement learning domains : A survey. Journal of Machine Learning Research, 10, 1633–1685.
36.
TaylorM.StoneP.LiuY. (2007). Transfer learning via inter-task mappings for temporal difference learning. Journal of Machine Learning Research, 8, 2125–2167.
37.
TorreyL.WalkerT.ShavlikJ.MaclinR. (2005). Using advice to transfer knowledge acquired in one reinforcement learning task to another. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer Berlin Heidelberg (pp. 412–424).
38.
TrigueroI.DerracJ.GarciaS.HerreraF. (2012). A taxonomy and experimental study on prototype generation for nearest neighbor classification. Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on, 42(1), 86–100.
39.
van HasseltH. (2011). Insights in reinforcement learning: formal analysis and empirical evaluation of temporal-difference learning algorithms (Unpublished doctoral dissertation). Universiteit Utrecht, The Netherlands.
40.
van OtterloM. (2009). The logic of adaptive behavior: Knowledge representation and algorithms for adaptive sequential decision making under uncertainty in first-order and relational domains. IOS Press.
