Damage recovery is critical for autonomous robots that need to operate for a long time without assistance. Most current methods are complex and costly because they require anticipating potential damage in order to have a contingency plan ready. As an alternative, we introduce the T-resilience algorithm, a new algorithm that allows robots to quickly and autonomously discover compensatory behavior in unanticipated situations. This algorithm equips the robot with a self-model and discovers new behavior by learning to avoid those that perform differently in the self-model and in reality. Our algorithm thus does not identify the damaged parts but it implicitly searches for efficient behavior that does not use them. We evaluate the T-resilience algorithm on a hexapod robot that needs to adapt to leg removal, broken legs and motor failures; we compare it to stochastic local search, policy gradient and the self-modeling algorithm proposed by Bongard et al. The behavior of the robot is assessed on-board thanks to an RGB-D sensor and a SLAM algorithm. Using only 25 tests on the robot and an overall running time of 20 min, T-resilience consistently leads to substantially better results than the other approaches.
ArgallBChernovaSVelosoM. (2009) A survey of robot learning from demonstration. Robotics and Autonomous Systems57(5): 469–483.
2.
BarfootTEaronED’EleuterioG (2006) Experiments in learning distributed control for a hexapod robot. Robotics and Autonomous Systems54(10): 864–872.
3.
BellinghamJGRajanK (2007) Robotics in remote and hostile environments. Science318(5853): 1098–1102.
4.
BerensonDEstevezNLipsonH (2005) Hardware evolution of analog circuits for in-situ robotic fault-recovery. In: Proceedings of NASA/DoD conference on evolvable hardware, pp. 12–19.
5.
BongardJ (2007) Action-selection and crossover strategies for self-modeling machines. In: Proceedings of genetic and evolutionary computation conference (GECCO), pp. 198–205.
6.
BongardJLipsonH (2005) Nonlinear system identification using coevolution of models and tests. IEEE Transactions on Evolutionary Computation9(4): 361–384.
7.
BongardJZykovVLipsonH (2006) Resilient machines through continuous self-modeling. Science314(5802): 1118–1121.
8.
CaccavaleFVillaniL (2002) Fault Diagnosis and Fault Tolerance for Mechatronic Systems: Recent Advances. New York, NY: Springer.
9.
Cantu-PazE (2000) Efficient and Accurate Parallel Genetic Algorithms. Norwell, MA: Kluwer Academic Publishers.
10.
ChangCLinC (2011) Libsvm: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology2(3): 27.
11.
ChernovaSVelosoM (2004) An evolutionary approach to gait learning for four-legged robots. In: Proceedings of IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 2562–2567.
12.
CluneJStanleyKPennockR. (2011) On the performance of indirect encoding across the continuum of regularity. IEEE Transactions on Evolutionary Computation15(3): 346–367.
13.
ConnellJMahadevanS (1993) Robot Learning. New York, NY: Springer.
14.
CorbatoF (2007) On building systems that will fail. ACM Turing Award Lectures34(9): 72–81.
15.
CullyAMouretJB (2013 a) Behavioral repertoire learning in robotics. In: Proceedings of genetic and evolutionary computation conference (GECCO). pp. 175–182.
De JongK (2006) Evolutionary Computation: A Unified Approach. Cambridge, MA: MIT Press.
18.
DebK (2001) Multi-Objective Optimization Using Evolutionary Algorithms. New York, NY: John Wiley and Sons.
19.
DebKPratapAAgarwalS. (2002) A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation6(2): 182–197.
20.
DelcomynF (1971) The locomotion of the cockroach Periplaneta americana. Journal of Experimental Biology54(2): 443–452.
21.
DingXWangZRovettaA. (2010) Locomotion analysis of hexapod robot. Proceedings of conference on climbing and walking robots (CLAWAR), pp. 291–310.
22.
DoncieuxSMouretJ-BBredecheN. (2011) Evolutionary robotics: Exploring new horizons. In: DoncieuxSBredècheNMouretJ-B (eds) New Horizons in Evolutionary Robotics: Extended Contributions from the 2009 EvoDeRob Workshop. New York, NY: Springer, pp. 3–25.
23.
EndresFHessJEngelhardN. (2012) An evaluation of the RGB-D SLAM system. In: Proceedings of the IEEE international conference on robotics and automation (ICRA).
24.
GoldbergKChenB (2001) Collaborative control of robot motion: Robustness to error. In: Proceedings of IEEE/RSJ international conference on intelligent robots and systems (IROS).
25.
GörnerMHirzingerG (2010) Analysis and evaluation of the stability of a biologically inspired, leg loss tolerant gait for six- and eight-legged walking robots. In: Proceedings of the IEEE international conference on robotics and automation (ICRA), pp. 4728–4735.
26.
GrefenstetteJJSchultzACMoriartyDE (1999) Evolutionary algorithms for reinforcement learning. Journal of Artificial Intelligence Research11: 241–276.
27.
HartlandCBredecheN (2006) Evolutionary robotics, anticipation and the reality gap. In: IEEE international conference on robotics and biomimetics, pp. 1640–1645.
28.
Heidrich-MeisnerVIgelC (2009) Neuroevolution strategies for episodic reinforcement learning. Journal of Algorithms64(4): 152–168.
29.
HemkerTStelzerMVon StrykO. (2009) Efficient walking speed optimization of a humanoid robot. The International Journal of Robotics Research28(2): 303–314.
30.
HoffmannMMarquesHArietaA. (2010) Body schema in robotics: A review. IEEE Transactions on Autonomous Mental Development2(4): 304–324.
31.
HollandOGoodmanR (2003) Robots with internal models: A route to machine consciousness?Journal of Consciousness Studies10(4–5): 4–5.
32.
HoosHHStützleT (2005) Stochastic Local Search: Foundations And Applications. Burlington, MA: Morgan Kaufmann.
33.
HornbyGTakamuraSYamamotoT. (2005) Autonomous evolution of dynamic gaits with two quadruped robots. IEEE Transactions on Robotics21(3): 402–410.
34.
HornbyGSLohnJDLindenDS (2011) Computer-automated evolution of an X-band antenna for NASA’s Space Technology 5 mission. Evolutionary Computation19(1): 1–23.
35.
JakimovskiBMaehleE (2010) In situ self-reconfiguration of hexapod robot OSCAR using biologically inspired approaches. In: MiripourB (ed.) Climbing and Walking Robots. Rijeka, Croatia: InTech.
36.
JakobiNHusbandsPHarveyI (1995) Noise and the reality gap: The use of simulation in evolutionary robotics. Proceedings of the European conference on artificial life (ECAL), pp. 704–720.
37.
KajitaSEspiauB (2008) Legged robots. In: SicilianoBKhatibO (eds) Handbook of Robotics. New York, NY: Springer, pp. 361–389.
38.
KatićDVukobratovićM (2003) Survey of intelligent control techniques for humanoid robots. Journal of Intelligent & Robotic Systems37(2): 117–141.
39.
KimuraHYamashitaTKobayashiS (2001) Reinforcement learning of walking behavior for a four-legged robot. In: Proceedings of IEEE conference on decision and control (CDC), pp. 411–416.
40.
KlausGGletteKTørresenJ (2012) A comparison of sampling strategies for parameter estimation of a robot simulator. Simulation, Modeling, and Programming for Autonomous Robots7628: 173–184.
41.
KoberJPetersJ (2010) Imitation and reinforcement learning – practical learning algorithms for motor primitives in robotics. IEEE Robotics and Automation Magazine17(2): 1–8.
42.
KoberJPetersJ (2012) Reinforcement learning in robotics: A survey. In: Adaptation, Learning, and Optimization (ALO). New York, NY: Springer, pp. 579–610.
43.
KohlNStoneP (2004) Policy gradient reinforcement learning for fast quadrupedal locomotion. In: Proceedings of the IEEE international conference on robotics and automation (ICRA), pp. 2619–2624.
44.
KoosSMouretJB (2011) Online discovery of locomotion modes for wheel-legged hybrid robots: A transferability-based approach. In: Proceedings of the conference on climbing and walking robots (CLAWAR), pp. 70–77.
45.
KoosSMouretJBDoncieuxS (2013) The transferability approach: Crossing the reality gap in evolutionary robotics. IEEE Transactions on Evolutionary Computation17(1): 22–145.
46.
KorenIKrishnaCM (2007) Fault-Tolerant Systems. Burlington, MA: Morgan Kaufmann.
47.
LinCMChenCH (2007) Robust fault-tolerant control for a biped robot using a recurrent cerebellar model articulation controller. Systems, Man, and Cybernetics, Part B: Cybernetics37(1): 110–123.
48.
MahdaviSBentleyP (2003) An evolutionary approach to damage recovery of robot motion with muscles. Advances in Artificial Life2801: 248–255.
MooreGE (1975) Progress in digital integrated electronics. In: International electron devices meeting, pp. 11–13.
53.
MostafaKTsaiCHerI (2010) Alternative gaits for multiped robots with leg failures to retain maneuverability. International Journal of Advanced Robotic Systems7(4): 31.
54.
MouretJBDoncieuxS (2010) Sferes: Evolvin’ in the multi-core world. In: Proceedings of the IEEE congress on evolutionary computation (CEC), pp. 4079–4086.
55.
MouretJBDoncieuxS (2012) Encouraging behavioral diversity in evolutionary robotics: An empirical study. Evolutionary Computation20(1): 91–133.
56.
MouretJBKoosSDoncieuxS (2012) Crossing the reality gap: A short introduction to the transferability approach. In: Proceedings of ALIFE’s workshop ‘Evolution in physical systems‘, pp. 1–7.
57.
MulderTHochstenbachJDijkstraP. (2008) Born to adapt, but not in your dreams. Consciousness and Cognition17(4): 1266–71.
58.
NakamuraYMoriTSatoM. (2007) Reinforcement learning for a biped robot based on a CPG-actor-critic method. Neural Networks20(6): 723–735.
59.
NelsonABarlowGDoitsidisL (2009) Fitness functions in evolutionary robotics: A survey and analysis. Robotics and Autonomous Systems57(4): 345–370.
60.
Nguyen-TuongDPetersJ (2011) Model learning for robot control: A survey. Cognitive Processing12(4): 319–340.
61.
PalmerMMillerDBlackwellT (2009) An evolved neural controller for bipedal walking: Transitioning from simulator to hardware. In: Proceedings of IROS workshop on exploring new horizons in evolutionary design of robots.
62.
ParkerG (2009) Punctuated anytime learning to evolve robot control for area coverage. Design and Control of Intelligent Robotic Systems177: 255–277.
PetersJSchaalS (2008) Reinforcement learning of motor skills with policy gradients. Neural Networks21(4): 682–697.
65.
PrasslerEKosugeK (2008) Domestic robotics. In: SicilianoBKhatibO (eds.) Handbook of Robotics. New York, NY: Springer, pp. 1253–1281.
66.
PretoriusCdu PlessisMCilliersC (2012) Simulating robots without conventional physics: A neural network approach. Journal of Intelligent & Robotic Systems71: 319–348.
67.
QuZIhlefeldCMJinY. (2003) Robust fault-tolerant self-recovering control of nonlinear uncertain systems. Automatica39(10): 1763–1771.
68.
QuigleyMConleyKGerkeyBP. (2009) ROS: An open-source robot operating system. In: Proceedings of ICRA’s workshop on open source software.
69.
RamachandranVHirsteinW (1998) The perception of phantom limbs. Brain121(9): 1603–1630.
70.
SaranliUBuehlerMKoditschekD (2001) Rhex: A simple and highly mobile hexapod robot. The International Journal of Robotics Research20(7): 616–631.
71.
SchleyerGRussellA (2010) Adaptable gait generation for autotomised legged robots. In: Proceedings of Australasian conference on robotics and automation (ACRA).
72.
SchmitzJDeanJKindermannT. (2001) A biologically inspired controller for hexapod walking: Simple solutions by exploiting physical properties. The Biological Bulletin200(2): 195–200.
73.
SmolaASchölkopfB (2004) A tutorial on support vector regression. Statistics and Computing14(3): 199–222.
74.
SmolaAVapnikV (1997) Support vector regression machines. Advances in Neural Information Processing Systems9: 155–161.
75.
SproewitzAMoeckelRMayeJ. (2008) Learning to move in modular robots using central pattern generators and online optimization. The International Journal of Robotics Research27(3–4): 423–443.
76.
SteingrubeSTimmeMWörgötterF. (2010) Self-organized adaptation of a simple neural circuit enables complex robot behaviour. Nature Physics6(3): 224–230.
77.
SturmJPlagemannCBurgardW (2008) Adaptive body scheme models for robust robotic manipulation. In: Robotics: Science and systems.
78.
SuttonRMcAllesterDSinghS. (2000) Policy gradient methods for reinforcement learning with function approximation. Advances In Neural Information Processing Systems12(22): 1057–1063.
79.
SuttonRSBartoAG (1998) Introduction to Reinforcement Learning. Cambridge, MA: MIT Press.
80.
TedrakeRZhangTSeungH (2005) Learning to walk in 20 minutes. In: Proceedings of Yale workshop on adaptive and learning systems.
81.
ToffoloABeniniE (2003) Genetic diversity as an objective in multi-objective evolutionary algorithms. Evolutionary Computation11(2): 151–167.
82.
TogeliusJSchaulTWierstraD. (2009) Ontogenetic and phylogenetic reinforcement learning. Kuenstliche Intelligenz23(3): 30–33.
83.
TuringAM (1950) Computing machinery and intelligence. Mind59(236): 433–460.
84.
VisinskyMCavallaroJWalkerI (1994) Robotic fault detection and fault tolerance: A survey. Reliability Engineering & System Safety46(2): 139–158.
85.
VogeleyKKurthenMFalkaiP. (1999) Essential functions of the human self model are implemented in the prefrontal cortex. Consciousness and Cognition8(3): 343–63.
86.
WeingartenJLopesGBuehlerM. (2004) Automated gait adaptation for legged robots. In: Proceedings of the IEEE international conference on robotics and automation (ICRA), pp. 2153–2158.
87.
WhitesonS (2012) Evolutionary computation for reinforcement learning. In: WieringMvan OtterloM (eds) Reinforcement Learning: State of the Art. Berlin: Springer, pp. 326–355.
88.
WilsonD (1966) Insect walking. Annual Review of Entomology11(1): 103–122.
89.
YosinskiJCluneJHidalgoD. (2011) Evolving robot gaits in hardware: The HyperNEAT generative encoding vs. parameter optimization. In: Proceedings of the 20th European conference on artificial life (ECAL), pp. 11–18.
90.
ZagalJDelpianoJRuiz-del SolarJ (2009) Self-modeling in humanoid soccer robots. Robotics and Autonomous Systems57(8): 819–827.
91.
ZagalJRuiz-del SolarJVallejosP (2004) Back to reality: Crossing the reality gap in evolutionary robotics. In: Proceedings of IFAC symposium on intelligent autonomous vehicles (IAV).
92.
ZykovV (2008) Morphological and behavioral resilience against physical damage for robotic systems. PhD Thesis, Cornell University, NY, USA.
93.
ZykovVBongardJLipsonH (2004) Evolving dynamic gaits on a physical robot. In: Proceedings of genetic and evolutionary computation conference, late breaking paper (GECCO).
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
