Imitation is an example of social learning in which an individual observes and copies another’s actions. This paper presents a new method for using imitation as a way of enhancing the learning speed of individual agents that employ a well-known reinforcement learning algorithm, namely Q-learning. Compared with other research that uses imitation with reinforcement learning, our method uses imitation of purely observed behaviours to enhance learning, with no internal state access or sharing of experiences between agents. The paper evaluates our imitation-enhanced reinforcement learning approach in both simulation and with real robots in continuous space. Both simulation and real robot experimental results show that the learning speed of the group is improved.
AbbeelP.NgA. Y. (2004). Apprenticeship learning via inverse reinforcement learning. Proceedings of ICML 2004.
2.
BartoA. G.BradtkeS. J.SinghS. P. (2004). Learning to act using real-time dynamic programming. Artificial Intelligence, 6, 105–122.
3.
BentivegnaD. C.AtkesonC. G.ChengG. (2004). Learning from observation and practice using behavioral primitives: marble maze. Robotics, 551–560.
4.
BillardA.CalinonS.DillmannR.SchaalS. (2008). Robot programming by demonstration. Springer Handbook of Robotics, 1371–1394.
5.
BradtkeS. J.DuffM. O. (1995). Reinforcement learning methods for continuous-time markov decision problems. In TesauroG.TouretzkyD.LeenT. (Eds.), Advances in Neural Information Processing Systems: Proceedings of the 1994 Conference (pp. 393–400). Cambridge, MA, USA: MIT Press.
6.
BreazealC.BuchsbaumD.GrayJ.GatenbyD.BlumbergB. (2005). Learning from and about others: Towards using imitation to bootstrap the social understanding of others by robots. Artificial Life, 11(1–2), 31–62.
7.
CalinonS.BillardA. (2007). Incremental learning of gestures by imitation in a humanoid robot. In Proceedings of ACM/IEEE International Conference on Human–Robot Interaction (pp. 255–262).
8.
CritesR. H.BartoA. G. (1996). Improving elevator performance using reinforcement learning. In TouretzkyD.MozerM.HasselmoM. (Eds.), Proceedings of the Neural Information Processing Systems 8.
9.
DillmannR. (2004). Teaching and learning of robot tasks via observation of human performance. Journal of Robotics and Autonomous Systems, 47(2–3), 109–116.
10.
DoyaK. (2000). Reinforcement learning in continuous time and space. Neural Computation, 12, 219–245.
11.
ErbasM. D.WinfieldA. F. T. (2011). On the emergence of structure in behaviours evolved through embodied imitation in group of robots. In Proceedings of the Eleventh European Conference on the Synthesis and Simulation of Living Systems (ECAL 2011) (pp. 219–226).
12.
GaussierP.MogaS.BanquetJ. P.QuoyM. (1998). From perception–action loop to imitation process: a bottom-up approach of learning by imitation. Applied Artificial Intelligence, 7(1), 701–729.
13.
GuenterF.BillardA. G. (2007). Using reinforcement learning to adapt an imitation task. Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, 1022–1027.
14.
LatzkeT.BehnkeS.BenewitzM. (2006). Imitative reinforcement learning for soccer playing robots. In Proceedings of 10th International RoboCup Symposium.
15.
LiuW.WinfieldA. F. T. (2011). Open-hardware e-puck linux extension board for experimental swarm robotics research. Microprocessors and Microsystems, 35(1).
16.
MondadaF.BonaniM.RaemyX.PughJ.CianciC.KlaptoczA.MagnenatS.ZuffereyJ. C.FloreanoD.MartinoliA. (2009). The e-puck, a robot designed for education in engineering. In 9th Conference on Autonomous Robot Systems and Competitions (pp. 59–65).
17.
NehanivC. L.DautenhahnK. (2002). The correspondence problem. In NehanivC. L.DautenhahnK. (Eds.), Imitation in animals amd artifacts (p. 41–61). MIT Press.
18.
NehanivC. L.DautenhahnK. (2007). Imitation and Social Learning in Robots, Humans and Animals. Cambridge University Press.
19.
NicolescuM.MataricM. J. (2007). Task learning through imitation and human–robot interaction. In NehanivC. L.DautenhahnK. (Eds.), Imitation and Social Learning in Robots, Humans and Animals (p. 407–424). Cambridge University Press.
20.
NolfiS.FloreanoD. (2000). Evolutionary Robotics. MIT Press.
21.
PetersJ.SchaalS. (2006). Policy gradient methods for robotics. In Proceedings of the International Conference on Artificial Neural Networks.
RumelhartD. E.HintonG. E.WilliamR. J. (1986). Learning representation by back-propagation. Nature, 323, 533–536.
24.
SantamariaJ. C.SuttonR. S.RamA. (1998). Experiments with reinforcement learning in problems with continuous state and action spaces. Adaptive Behaviour, 6(2), 163–218.
25.
SmartW. D.KaelblingL. P. (2000). Practical reinforcement learning in continuous spaces. In Proceedings of the 17th International Conference on Machine Learning (pp. 903–910).
26.
StrosslinT.GerstnerW. (2006). Reinforcement learning in continuous state and action space. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems.
27.
SuttonR. S.BartoG. B. (1998). Reinforcement learning. MIT Press.
