Vision-based reinforcement learning (RL) is a generalizable way to control an agent because it is agnostic of specific hardware configurations. As visual observations are highly entangled, attempts for vision-based RL rely on scene representation that discerns individual entities and establishes intuitive physics to constitute the world model. However, most existing works on scene representation learning cannot successfully be deployed to train an RL agent, as they are often highly unstable and fail to sustain for a long enough temporal horizon. We propose ASIMO, a fully unsupervised scene decomposition to perform interaction-rich tasks with a vision-based RL agent. ASIMO decomposes agent-object interaction videos of episodic-length into the agent, objects, and background, predicting their long-term interactions. Further, we explicitly model possible occlusion in the image observations and stably track individual objects. Then, we can correctly deduce the updated positions of individual entities in response to the agent action, only from partial visual observation. Based on the stable entity-wise decomposition and temporal prediction, we formulate a hierarchical framework to train the RL agent that focuses on the context around the object of interest. We demonstrate that our formulation for scene representation can be universally deployed to train different configurations of agents and accomplish several tasks that involve pushing, arranging, and placing multiple rigid objects.
BearDMFanCMrowcaD, et al. (2020) Learning Physical Graph Representations from Visual Scenes. arXiv preprint arXiv:2006.12373.
2.
BuesingLWeberTRacaniereS, et al. (2018) Learning and Querying Fast Generative Models for Reinforcement Learning. arXiv preprint arXiv:1802.03006.
3.
BurgessCPMattheyLWattersN, et al. (2019) Monet: Unsupervised Scene Decomposition and Representation. arXiv preprint arXiv:1901.11390.
4.
CabiSColmenarejoSGNovikovA, et al. (2019) Scaling Data-Driven Robotics with Reward Sketching and Batch Reinforcement Learning. arXiv preprint arXiv:1909.12200.
5.
CaronMTouvronHMisraI, et al. (2021) Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, 10–17 October 2021, 9650–9660.
6.
ChenBAbbeelPPathakD (2021) Unsupervised Learning of Visual 3d Keypoints for Control. arXiv preprint arXiv:2106.07643.
7.
ChenHVenkateshRFriedmanY, et al. (2022) Unsupervised segmentation in real-world images via spelke object inference. In European Conference on Computer Vision. Berlin: Springer, 719–735.
8.
ChungJKastnerKDinhL, et al. (2015) A recurrent latent variable model for sequential data. Advances in Neural Information Processing Systems. Montréal, Canada: 28th Annual Conference, 2980–2988.
CrawfordEPineauJ (2019b) Spatially invariant unsupervised object detection with convolutional neural networks. Proceedings of the AAAI Conference on Artificial Intelligence33: 3412–3420.
DengJGuoJXueN, et al. (2018) Arcface: additive angular margin loss for deep face recognition. In: 2019 IEEE CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, 15–20 June 2019, 4685–4694.
DuYSmithKUlmanT, et al. (2020) Unsupervised Discovery of 3d Physical Objects from Video. arXiv preprint arXiv:2007.12348.
17.
EbertFFinnCLeeAX, et al. (2017) Self-Supervised Visual Planning With Temporal Skip Connections. arXiv preprint arXiv:1710.05268.
18.
EbertFFinnCDasariS, et al. (2018) Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control. arXiv preprint arXiv:1812.00568.
19.
ElsayedGMahendranAvan SteenkisteS, et al. (2022) Savi++: towards end-to-end object-centric learning from real-world videos. Advances in Neural Information Processing Systems35: 28940–28954.
20.
EmamiPHePRankaS, et al. (2021) Efficient Iterative Amortized Inference for Learning Symmetric and Disentangled Multi-Object Representations. arXiv preprint arXiv:2106.03630.
21.
EngelckeMKosiorekARJonesOP, et al. (2019) Genesis: Generative Scene Inference and Sampling with Object-Centric Latent Representations. arXiv preprint arXiv:1907.13052.
22.
EngelckeMParker JonesOPosnerI (2021) Genesis-v2: inferring unordered object representations without iterative refinement. Advances in Neural Information Processing Systems34: 8085–8094.
23.
EslamiSAHeessNWeberT, et al. (2016) Attend, infer, repeat: fast scene understanding with generative models. Advances in Neural Information Processing Systems. Barcelona, Spain: 29th Annual Conference, 3225–3233.
24.
FranceschiJYDelasallesEChenM, et al. (2020) Stochastic latent residual video prediction. In: International Conference on Machine Learning PMLR, 37th Annual Conference,Vienna, Austria, 3233–3246.
25.
FujimotoSVan HoofHMegerD (2018) Addressing Function Approximation Error in Actor-Critic Methods. arXiv preprint arXiv:1802.09477.
26.
GazCCognettiMOlivaA, et al. (2019) Dynamic identification of the Franka Emika panda robot with retrieval of feasible parameters using penalty-based optimization. IEEE Robotics and Automation Letters4(4): 4147–4154.
27.
GehringJSynnaeveGKrauseA, et al. (2021) Hierarchical skills for efficient exploration. Advances in Neural Information Processing Systems34: 11553–11564.
28.
GreffKKaufmanRLKabraR, et al. (2019) Multi-object Representation Learning with Iterative Variational Inference. arXiv preprint arXiv:1903.00450.
29.
GreffKvan SteenkisteSSchmidhuberJ (2020) On the Binding Problem in Artificial Neural Networks. arXiv preprint arXiv:2012.05208.
30.
GregorKRezendeDJBesseF, et al. (2019) Shaping belief states with generative environment models for rl. Advances in Neural Information Processing Systems. Vancouver, Canada: 33rd Annual Conference, 13475–13487.
31.
HaarnojaTZhouAHartikainenK, et al. (2018) Soft Actor-Critic Algorithms and Applications. arXiv preprint arXiv:1812.05905.
32.
HafnerDLillicrapTFischerI, et al. (2018) Learning Latent Dynamics for Planning from Pixels. arXiv preprint arXiv:1811.04551.
33.
HafnerDLillicrapTBaJ, et al. (2019) Dream to Control: Learning Behaviors by Latent Imagination. arXiv preprint arXiv:1912.01603.
34.
HafnerDLillicrapTNorouziM, et al. (2020) Mastering Atari with Discrete World Models. arXiv preprint arXiv:2010.02193.
35.
HansenNJangirRSunY, et al. (2020) Self-supervised Policy Adaptation during Deployment. arXiv preprint arXiv:2007.04309.
36.
HeraviNWahidALynchC, et al. (2023) Visuomotor control in multi-object scenes using object-aware representations. In: 2023 IEEE International Conference on Robotics and Automation (ICRA), London, 29 May 2023–02 June 2023, 9515–9522. IEEE.
37.
HuangSOntañónS (2020) A Closer Look at Invalid Action Masking in Policy Gradient Algorithms. arXiv preprint arXiv:2006.14171.
38.
HuangTYangGTangG (1979) A fast two-dimensional median filtering algorithm. IEEE Transactions on Acoustics, Speech, & Signal Processing27(1): 13–18.
39.
JaderbergMSimonyanKZissermanA, et al. (2015) Spatial transformer networks. Advances in Neural Information Processing Systems. Montréal, Canada: 28th Annual Conference, 2017–2025.
40.
JakabTGuptaABilenH, et al. (2018) Unsupervised learning of object landmarks through conditional image generation. In: Advances in Neural Information Processing Systems (NeurIPS), 31st Annual Conference,Montréal, Canada, 4016-4027.
41.
JamesSMaZArrojoDR, et al. (2020) Rlbench: the robot learning benchmark & learning environment. IEEE Robotics and Automation Letters5(2): 3019–3026.
42.
JiangJJanghorbaniSDe MeloG, et al. (2020) Scalor: generative world models with scalable object representations. In: Proceedings of the International Conference on Learning Representations (ICLR), 8th Annual Conference. Virtual.
43.
KabraRZoranDErdoganG, et al. (2021) Simone: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition. arXiv preprint arXiv:2106.03849.
44.
KarazijaLLainaIRupprechtC (2021) Clevrtex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentation. arXiv preprint arXiv:2111.10265.
45.
KimTAhnSBengioY (2019) Variational temporal abstraction. Advances in Neural Information Processing Systems32: 11570–11579.
KingmaDPBaJAAdamJ (2020) A Method for Stochastic Optimization. arxiv 2014. arXiv preprint arXiv:1412.6980 106.
48.
KipfTElsayedGFMahendranA, et al. (2021) Conditional Object-Centric Learning from Video. arXiv preprint arXiv:2111.12594.
49.
KosiorekAKimHTehYW, et al. (2018) Sequential attend, infer, repeat: generative modelling of moving objects. Advances in Neural Information Processing Systems. Montréal, Canada: 32nd Annual Conference, 8606–8616.
50.
KrishnanRShalitUSontagD (2017) Structured inference networks for nonlinear state space models. In: Proceedings of the AAAI Conference on Artificial Intelligence, 31st Annual Conference,San Francisco, CA, USA. 2101–2109.
51.
KumarAZhouATuckerG, et al. (2020) Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems33: 1179–1191.
52.
LiNRazaMAHuW, et al. (2021) Object-centric representation learning with generative spatial-temporal factorization. Advances in Neural Information Processing Systems. Virtual: 34th Annual Conference, 34.
53.
LiangELiawRNishiharaR, et al. (2018) Rllib: abstractions for distributed reinforcement learning. In: Proceedings of the International Conference on Machine Learning (ICML), 35th Annual Conference.Stockholm, Sweden, PMLR, 3053–3062.
54.
LinZWuYFPeriS, et al. (2020a) Improving Generative Imagination in Object-Centric World Models. arXiv preprint arXiv:2010.02054.
55.
LinZYangDZhaoL, et al. (2020b) Rd2: reward decomposition with representation decomposition. Advances in Neural Information Processing Systems33: 11298–11308.
56.
LiuCXuXHuD (2014) Multiobjective reinforcement learning: a comprehensive overview. IEEE Transactions on Systems, Man, and Cybernetics: Systems45(3): 385–398.
57.
LocatelloFWeissenbornDUnterthinerT, et al. (2020) Object-Centric Learning With Slot Attention. arXiv preprint arXiv:2006.15055.
58.
MaddisonCJMnihATehYW (2016) The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables. arXiv preprint arXiv:1611.00712.
59.
MakoviychukVWawrzyniakLGuoY, et al. (2021) Isaac Gym: High Performance Gpu-Based Physics Simulation for Robot Learning. arXiv preprint arXiv:2108.10470.
60.
McCarthyADLiXGuJ, et al. (2020) Addressing posterior collapse with mutual information for improved variational neural machine translation. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.851–861.
61.
MendoncaRRybkinODaniilidisK, et al. (2021) Discovering and achieving goals via world models. Advances in Neural Information Processing Systems. Virtual: 35th Annual Conference, 34.
62.
MinCHBaeJLeeJ, et al. (2021) Gatsbi: generative agent-centric spatio-temporal object interaction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, 20–25 June 2021, 3074–3083.
63.
MindererMSunCVillegasR, et al. (2019) Unsupervised learning of object structure and dynamics from videos. Advances in Neural Information Processing Systems. Vancouver, Canada: 33rd Annual Conference, 92–102.
64.
NachumOGuSLeeH, et al. (2018a) Near-optimal Representation Learning for Hierarchical Reinforcement Learning. arXiv preprint arXiv:1810.01257.
65.
NachumOGuSSLeeH, et al. (2018b) Data-efficient hierarchical reinforcement learning. In Advances in Neural Information Processing Systems. 31st Annual Conference,Montréal, Canada, 31.
66.
NairSFinnC (2019) Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks Via Visual Subgoal Generation. arXiv preprint arXiv:1909.05829.
67.
NwankpaCIjomahWGachaganA, et al. (2018) Activation Functions: Comparison of Trends in Practice and Research for Deep Learning. arXiv preprint arXiv:1811.03378.
68.
PaszkeAGrossSMassaF, et al. (2019) Pytorch: an imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems. Vancouver, Canada: 33rd Annual Conference, 32.
69.
PathakDAgrawalPEfrosAA, et al. (2017) Curiosity-driven exploration by self-supervised prediction. In: International Conference on Machine Learning PMLR, 34th Annual Conference,Sydney, Australia, 2778–2787.
70.
PertschKRybkinOEbertF, et al. (2020) Long-Horizon Visual Planning With Goal-Conditioned Hierarchical Predictors. arXiv preprint arXiv:2006.13205.
71.
RezazadehABadithelaADesinghK, et al. (2023) Slotgnn: Unsupervised Discovery of Multi-Object Representations and Visual Dynamics. arXiv preprint arXiv:2310.04617.
72.
RohmerESinghSPFreeseM (2013) Coppeliasim (formerly v-rep): a versatile and scalable robot simulation framework. In: Proceedings of the International Conference on Intelligent Robots and Systems (IROS),Tokyo, Japan, 1321–1326.
73.
SajjadiMSDuckworthDMahendranA, et al. (2022) Object scene representation transformer. Advances in Neural Information Processing Systems35: 9512–9524.
74.
SaxenaVBaJHafnerD (2021) Clockwork variational autoencoders. Advances in Neural Information Processing Systems. Virtual: 35th Annual Conference, 34.
SeitzerMSchölkopfBMartiusG (2021) Causal influence detection for improving efficiency in reinforcement learning. Advances in Neural Information Processing Systems34: 22905–22918.
77.
SeitzerMHornMZadaianchukA, et al. (2022) Bridging the gap to real-world object-centric learning. In: The Eleventh International Conference on Learning Representations, Virtual, 1234–1245.
78.
SeoYLeeKJamesS, et al. (2022) Reinforcement Learning With Action-Free Pre-Training from Videos. arXiv preprint arXiv:2203.13880.
79.
SeoYKimJJamesS, et al. (2023) Multi-View Masked World Models for Visual Robotic Manipulation. arXiv preprint arXiv:2302.02408.
80.
SinghGDengFAhnS (2021a) Illiterate Dall-E Learns to Compose. arXiv preprint arXiv:2110.11405.
81.
SinghGPeriSKimJ, et al. (2021b) Structured world belief for reinforcement learning in pomdp. In: International Conference on Machine Learning PMLR, 38th Annual Conference, Virtual,9765–9776.
82.
SinghGWuYFAhnS (2022) Simple unsupervised object-centric learning for complex and naturalistic videos. Advances in Neural Information Processing Systems35: 18181–18196.
83.
SohnKLeeHYanX (2015) Learning structured output representation using deep conditional generative models. Advances in Neural Information Processing Systems. Montréal, Canada: 28th Annual Conference, 3483–3491.
StelznerKPeharzRKerstingK (2019) Faster attend-infer-repeat with tractable probabilistic models. In: International Conference on Machine Learning, 36th Annual Conference,Long Beach, CA, USA, PMLR, 5966–5975.
86.
SuttonRSBartoAG (2018) Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press.
87.
UnterthinerTvan SteenkisteSKurachK, et al. (2018) Towards Accurate Generative Models of Video: A New Metric & Challenges. arXiv preprint arXiv:1812.01717.
88.
VaswaniAShazeerNParmarN, et al. (2017) Attention is all you need. Advances in Neural Information Processing Systems. Long Beach, CA, USA: 30th Annual Conference, 5998–6008.
89.
VeerapaneniRCo-ReyesJDChangM, et al. (2019) Entity Abstraction in Visual Model-Based Reinforcement Learning. arXiv preprint arXiv:1910.12827.
90.
WangZBovikACSheikhHR, et al. (2004) Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing: A Publication of the IEEE Signal Processing Society13(4): 600–612.
91.
WatterMSpringenbergJBoedeckerJ, et al. (2015) Embed to control: a locally linear latent dynamics model for control from raw images. Advances in Neural Information Processing Systems. Montréal, Canada: 28th Annual Conference, 2746–2754.
92.
WattersNMattheyLBosnjakM, et al. (2019) Cobra: Data-Efficient Model-Based Rl through Unsupervised Object Discovery and Curiosity-Driven Exploration. arXiv preprint arXiv:1905.09275.
93.
WeisMAChittaKSharmaY, et al. (2020) Unmasking the Inductive Biases of Unsupervised Object Representations for Video Sequences. arXiv preprint arXiv:2006.07034.
94.
WuYJonesOPEngelckeM, et al. (2021) Apex: Unsupervised, Object-Centric Scene Segmentation and Tracking for Robot Manipulation. arXiv preprint arXiv:2105.14895.
95.
WuZDvornikNGreffK, et al. (2022) Slotformer: Unsupervised Visual Dynamics Simulation with Object-Centric Models. arXiv preprint arXiv:2210.05861.
96.
WulfmeierMByravanAHertweckT, et al. (2020) Representation Matters: Improving Perception and Exploration for Robotics. arXiv preprint arXiv:2011.01758.
97.
YoonJWuYFBaeH, et al. (2023) An Investigation into Pre-Training Object-Centric Representations for Reinforcement Learning. arXiv preprint arXiv:2302.04419.
98.
YuTQuillenDHeZ, et al. (2020) Meta-world: a benchmark and evaluation for multi-task and meta reinforcement learning. In: Proceedings of the Conference on Robot Learning (CoRL),Cambridge, MA, USA, PMLR, 211–223.
99.
YuHZhangHXuW (2022) Do You Need the Entropy Reward (in Practice)?arXiv preprint arXiv:2201.12434.
ZhengXZaheerMAhmedA, et al. (2017) State Space Lstm Models with Particle Mcmc Inference. arXiv preprint arXiv:1711.11179.
102.
ZoranDKabraRLerchnerA, et al. (2021) Parts: unsupervised segmentation with slots, attention and independence maximization. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, 10–17 October 2021, 10439–10447.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
