We propose an integrated learning by experience and demonstration algorithm that operates on the basis of both an objective scalar measure of performance and a demonstrated behaviour. The application of the method to two qualitatively different experimental scenarios involving simulated mobile robots demonstrates its efficacy. Indeed, the analysis of the obtained results shows that the robots trained through this integrated algorithm develop solutions that are functionally better than those obtained by using either a pure learning by demonstration, or a pure learning by experience algorithm. This is because the algorithm drives the learning process toward solutions that are qualitatively similar to the demonstration, but leaves the learning agent free to differentiate from the demonstration when this turns out to be necessary to maximize performance.
AbbeelP.NgA. Y. (2005). Exploration and apprenticeship learning in reinforcement learning. In Proceedings of the 22nd International Conference on Machine Learning. New York: ACP Press.
2.
ArgallB. D.BrowningB.VelosoM. (2010). Mobile robot motion control from demonstration and corrective feedback. In: SigaudO.PetersJ. (Eds.), From motor learning to interaction learning in robots, SCI (vol. 264, pp. 431–450). Heidelberg: Springer.
3.
ArgallB. D.ChernovaS.Veloso.M.BrowningB. (2009). A survey of robot learning from demonstration. Robotics and Autonomous Systems, 57(5), 469–483.
4.
ArgallB. D.BrowningB.VelosoM. (2008). Learning robot motion control with demonstration and advice-operators. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS’08) (pp. 399–404). IEEE Press.
5.
BillardA.CallinonS.DillmannR.SchaalS. (2008). Robot programming by demonstration. In SicilianoB.KhatibO. (Eds.), Handbook of robotics. New York: Springer.
6.
BillardA.CalinonS.GuenterF. (2006). Discriminative and adaptive imitation in uni-manual and bi-manual tasks. In The Social Mechanisms of Robot Programming by Demonstration, Robotics and Autonomous Systems, 54(5), 370–384 (special issue).
7.
BonaniM.LongchampV.MagnenatS.RétornazP.BurnierD.RouletG.MondadaF. (2010). The marXbot, a miniature mobile robot opening new perspectives for the collective-robotic research. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 4187–4193). Piscataway, NJ: IEEE Press.
8.
CetinaV. U. (2007). Supervised reinforcement learning using behavior models. In Sixth International Conference on Machine Learning and Applications. New York: IEEE Press.
9.
ChernovaS.VelosoM. (2007). Confidence-based learning from demonstration using Gaussian Mixture Models. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems (AAMAS’07) (pp. 1–8). ACM New York.
10.
CoatesA.AbbeelP.NgA. Y. (2008). Learning for control from multiple demonstrations. Proceedings of the 25th International Conference on Machine Learning (ICML ’08) (pp. 144–151). ACM New York.
11.
GlasmachersT.SchaulT.YiS.WierstraD.SchmidhuberJ. (2010). Exponential natural evolution strategies. In: Proceedings of the 12th Annual Conference on Genetic and Evolutionary Computation (GECCO’10), Portland, OR, pp. 393–400. New York: ACM.
12.
DanielC.NeumannG.PetersJ. (2012). Hierarchical relative entropy policy search. In LawrenceN.GirolamiM. (Eds), Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS 2012) (pp. 273–281).
13.
De GreefJ.NolfiS. (2010). Evolution of implicit and explicit communication in a group of mobile robots. In NolfiS.MirolliM. (Eds.), Evolution of communication and language in embodied agents. Berlin: Springer.
14.
HayesG.DemirisJ. (1994). A robot controller using learning by imitation. In BorkowskiA.CrowleyJ.L (Eds.), Proceedings of the 2nd International Symposium on Intelligent Robotic Systems (pp. 198–204). IEEE Press.
15.
HollandJ. H. (1975). Adaptation in natural and artificial systems. Ann Arbor, MI: University of Michigan Press.
16.
IjspeertA.NakanishiJ.SchaalS. (2004). Learning attractor landscapes for learning motor primitives. In ThrunS.SaulL. K.ScholkopfB. (Eds.), Advances in neural information processing systems (vol. 16). Cambridge, MA: MIT Press.
17.
JudahK.RoyS.FernA.DietterichT.G. (2010). Reinforcement learning via practice and critique advice. In Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence. Atlanta, GA: AAAI Press.
18.
KnoxW. B.StoneP. (2009). Interactively shaping agents via human reinforcement: The TAMER framework. In Proceedings of the Fifth International Conference on Knowledge Capture. New York: ACM Press.
19.
KnoxW. B.StoneP. (2010). Combining manual feedback with subsequent MDP reward signals for reinforcement learning. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems (vol. 1). Richland SC: International Foundation for Autonomous Agents and Multiagent Systems.
20.
KoberJ.BagnellJ. A.PetersJ. (2013). Reinforcement learning in robotics: A survey. The International Journal of Robotics Research32(11), 1238–127.
21.
KoberJ.PetersJ. (2009). Policy search for motor primitives in robotics. In KollerD.SchuurmansD.BengioY.BottouL. (Eds.), Advances in neural information processing systems 21. Carmel, IN: Current Associates Inc.
22.
MasseraG.FerrautoT.GigliottaO.NolfiS. (2013). FARSA: An open software tool for embodied cognitive science. In LiòP.MiglinoO.NicosiaG.NolfiS.PavoneM. (Eds.), Proceeding of the 12th European Conference on Artificial Life. Cambridge, MA: MIT Press.
23.
MondadaF.FranziE.IenneP. (1993). Mobile robot miniaturisation: A tool for investigation in control algorithms. In: Proceedings of the third International Symposium on Experimental Robotics, Kyoto, Japan
24.
NolfiS.FloreanoD. (2000). Evolutionary robotics: The biology, intelligence, and technology of self-organizing machines. Cambridge, MA: MIT Press/Bradford Books.
25.
ParaschosA.DanielC.PetersJ.NeumannG. (2013). Probabilistic movement primitives. In BurgesC. J. C.BottouL.WellingM.GhahramaniZ.WeinbergerK. Q. (Eds.), Advances in neural information processing systems (NIPS). Cambridge, MA: MIT Press
26.
RechenbergI.EigenM. (1973). Evolutionsstrategie: Optimierung technischer Systeme nach Prinzipien der biologischen Evolution. Stuttgart: Frommann-Holzboog.
27.
RosensteinM. T.BartoA. G. (2004). Supervised actor-critic reinforcement learning. In SiJ.BartoA.PowellW.WunschD. (Eds.), Learning and approximate dynamic programming: Scaling up to the real world. New York: John Wiley & Sons, Inc.
28.
RumelhartD. E.HintonG. E.WilliamsR. J. (1986). Learning internal representations by error propagation. In RumelhartD. E.McCellandI. L. (Eds.), Parallel distributed processing: Explorations in the microstructure of cognition (vol. 1). Cambridge, MA: MIT Press.
29.
RussellS.NorvigP. (2009). Artificial intelligence: A modern approach. Englewood Cliffs, NJ: Prentice Hall.
30.
SchaalS. (1997). Learning from demonstration. In: MozerM.C.JordanM.PetscheT. (Eds.), Advances in neural information processing systems 9. Cambridge, MA: MIT Press.
31.
SchaalS.MohajerianP.IjspeertA. (2007). Dynamics systems vs. optimal control—A unifying view. Progress in Brain Research, 165(1), 425–445.
32.
SchwefelH. P. (1977). Numerische Optimierung von Computer-Modellen mittels der Evolutionsstrategie (vol. 26). Basel/Stuttgart: Birkhaeuser.
33.
SchwefelH. P. (1995), Evolution and optimum seeking. New York: Wiley.
34.
SperatiV.TrianniV.NolfiS. (2008). Evolving coordinated group behaviour through maximization of mean mutual information. Special Issue on Swarm Robotics, Swarm Intelligence Journal, 4(2), 73–95.
35.
SuttonR. S.BartoA. G. (1998). Reinforcement learning. Boston, MA: MIT Press.
36.
TaylorM.SuayH.ChernovaS. (2011). Integrating reinforcement learning with human demonstrations of varying ability, AAMAS, Taipei, Taiwan.
37.
ValentiM. (2013). Learning of manipulation capabilities in a humanoid robot. Master thesis, School of Engineering, Università degli Studi di Roma ‘La Sapienza’, Italy.
38.
WierstraD.SchaulT.PetersJ.SchmidhuberJ. (2008). Natural evolution strategies. In Proceedings of the Congress on Evolutionary Computation (CEC08). Hong Kong: IEEE Press.
