Reinforcement learning (RL) has been used to successfully solve sequential decision problem. However, considering risk at the same time as the learning process is an open research problem. In this work, we are interested in the type of risk that can lead to a catastrophic state. Related works that aim to deal with risk propose complex models. In contrast, we follow a simple, yet effective, idea: similar states might lead to similar risk. Using this idea, we propose risk mapping by similarity (RMS), an algorithm for discrete scenarios which infers the risk of newly discovered states by analyzing how similar they are to previously known risky states. In general terms, the RMS algorithm transfers the knowledge gathered by the agent regarding the risk to newly discovered states. We contribute with a new approach to consider risk based on similarity and with RMS, which is simple and generalizable as long as the premise similar states yield similar risk holds. RMS is not an RL algorithm, but a method to generate a risk-aware reward shaping signal that can be used with a RL algorithm to generate risk-aware policies.
AbbeelP.CoatesA.HunterT.NgA. (2009). Autonomous autorotation of an RC helicopter. In KhatibO.KumarV.PappasG. J. (Eds.), Experimental robotics (pp. 385–394). Berlin, Germany: Springer.
2.
AkametaluA. K.KaynamaS.FisacJ. F.ZeilingerM. N.GillulaJ. H.TomlinC. J. (2014). Reachability-based safe learning with Gaussian processes. In 2014 IEEE 53rd annual conference on decision and control (CDC) (pp. 1424–1431). New York, NY: IEEE.
3.
BlackP. E. (2006). Manhattan distance. Dictionary of Algorithms and Data Structures, 18, Article 2012.
4.
BoriahS.ChandolaV.KumarV. (2008). Similarity measures for categorical data: A comparative evaluation. In ApteC.ParkH.WangK.ZakiM. J. (Eds.), Proceedings of the 2008 SIAM international conference on data mining (pp. 243–254). Philadelphia, PA: SIAM.
5.
BrockmanG.CheungV.PetterssonL.SchneiderJ.SchulmanJ.TangJ.ZarembaW. (2016). OpenAI gym. Retrieved from https://arxiv.org/abs/1606.01540
FultonN.PlatzerA. (2018, February2–7). Safe reinforcement learning via formal methods. Paper presented at Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, LA.
8.
GarcíaJ.FernándezF. (2015). A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16, 1437–1480.
9.
GeibelP.WysotzkiF. (2005). Risk-sensitive reinforcement learning applied to control under constraints. Journal of Artificial Intelligence Research, 24, 81–108.
10.
GrześM. (2017). Reward shaping in episodic reinforcement learning. In LarsonK.WinikoffM.DasS.DurfeeE. (Eds.), Proceedings of the 16th conference on autonomous agents and multiagent systems (pp. 565–573). Richland, SC: International Foundation for Autonomous Agents and Multiagent Systems.
11.
HegerM. (1994). Consideration of risk in reinforcement learning. In CohenW. W.HirshH. (Eds.), Proceedings of the eleventh international conference on machine learning (pp. 105–111). San Francisco, CA: Morgan Kaufmann.
KahnG.VillaflorA.PongV.AbbeelP.LevineS. (2017). Uncertainty-aware reinforcement learning for collision avoidance (arXiv: 1702.01182). Retrieved from https://arxiv.org/abs/1702.01182
14.
KoberJ.BagnellJ. A.PetersJ. (2013). Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32, 1238–1274.
15.
LazaricA.RestelliM.BonariniA. (2008). Transfer of samples in batch reinforcement learning. In CohenW.McCalluA.RoweisS. (Eds.), Proceedings of the 25th international conference on machine learning (pp. 544–551). New York, NY: ACM.
16.
LeikeJ.MarticM.KrakovnaV.OrtegaP. A.EverittT.LefrancqA.LeggS. (2017). AI safety gridworlds (arXiv: 1711.09883). Retrieved from https://arxiv.org/abs/1711.09883
17.
LiptonZ. C.AzizzadenesheliK.KumarA.LiL.GaoJ.DengL. (2016). Combating reinforcement learning’s Sisyphean curse with intrinsic fear (arXiv: 1611.01211). Retrieved from https://arxiv.org/abs/1611.01211
18.
MahadevanS.MaggioniM. (2007). Proto-value functions: A laplacian framework for learning representation and control in markov decision processes. Journal of Machine Learning Research, 8, 2169–2231.
MnihV.KavukcuogluK.SilverD.GravesA.AntonoglouI.WierstraD.RiedmillerM. (2013). Playing atari with deep reinforcement learning (arXiv: 1312.5602). Retrieved from https://arxiv.org/abs/1312.5602
21.
MnihV.KavukcuogluK.SilverD.RusuA. A.VenessJ.BellemareM. G.… HassabisD. (2015). Human-level control through deep reinforcement learning. Nature, 518, 529–533.
22.
MorimuraT.SugiyamaM.KashimaH.HachiyaH.TanakaT. (2012). Parametric return density estimation for reinforcement learning (arXiv: 1203.3497). Retrieved from https://arxiv.org/abs/1203.3497
23.
Munoz de CoteE.GarciaE. O.MoralesE. F. (2016). Transfer learning by prototype generation in continuous spaces. Adaptive Behavior, 24, 464–478.
24.
NgA. Y.HaradaD.RussellS. (1999). Policy invariance under reward transformations: Theory and application to reward shaping. In BratkoI.DzeroskiS. (Eds.), Proceedings of the sixteenth international conference on machine learning (ICML) (pp. 278–287). San Francisco, CA: Morgan Kaufmann.
PendrithM. D. (2000). Distributed reinforcement learning for a traffic engineering application. In SierraC.GiniM.RosenscheinJ. S. (Eds.), Proceedings of the fourth international conference on autonomous agents (pp. 404–411). New York, NY: ACM.
27.
PeterJ. P.RyanM. J. (1976). An investigation of perceived risk at the brand level. Journal of Marketing Research, 13, 184–188.
28.
ResnickC.EldridgeW.HaD.BritzD.FoersterJ.TogeliusJ.BrunaJ. (2018). Pommerman: A multi-agent playground (arXiv: 1809.07124). Retrieved from https://arxiv.org/abs/1809.07124
29.
SuttonR. S.BartoA. G. (1998). Reinforcement learning: An introduction (Vol. 1). Cambridge, MA: The MIT press.
30.
TaylorM. E.JongN. K.StoneP. (2008). Transferring instances for model-based reinforcement learning. In DaelemansW.GoethalsB.MorikK. (Eds), Joint European conference on machine learning and knowledge discovery in databases (pp. 488–505). Berlin, Germany: Springer.
