Modeling and manipulating elasto-plastic objects are essential capabilities for robots to perform complex industrial and household interaction tasks (e.g., stuffing dumplings, rolling sushi, and making pottery). However, due to the high degrees of freedom of elasto-plastic objects, significant challenges exist in virtually every aspect of the robotic manipulation pipeline, for example, representing the states, modeling the dynamics, and synthesizing the control signals. We propose to tackle these challenges by employing a particle-based representation for elasto-plastic objects in a model-based planning framework. Our system, RoboCraft, only assumes access to raw RGBD visual observations. It transforms the sensory data into particles and learns a particle-based dynamics model using graph neural networks (GNNs) to capture the structure of the underlying system. The learned model can then be coupled with model predictive control (MPC) algorithms to plan the robot’s behavior. We show through experiments that with just 10 min of real-world robot interaction data, our robot can learn a dynamics model that can be used to synthesize control signals to deform elasto-plastic objects into various complex target shapes, including shapes that the robot has never encountered before. We perform systematic evaluations in both simulation and the real world to demonstrate the robot’s manipulation capabilities.
AntonovaRShiPYinH, et al. (2021a) Dynamic environments with deformable objects. In: Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, Virtual, December 6-14.
2.
AntonovaRVaravaAShiP, et al. (2021b) Sequential topological representations for predictive models of deformable objects. In: Learning for Dynamics and Control, Palo Alto, United States, June 23-24. PMLR, pp. 348–360.
3.
AntonovaRYangJSundaresanP, et al. (2022) A bayesian treatment of real-to-sim for deformable object manipulation. IEEE Robotics and Automation Letters7(3): 5819–5826.
4.
AtteneM (2010) A lightweight approach to repairing digitized polygon meshes. The Visual Computer26(11): 1393–1406.
5.
AvigalYBerscheidLAsfourT, et al. (2022) Speedfolding: learning efficient bimanual folding of garments. In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, pp. 1–8.
6.
BattagliaPPascanuRLaiM, et al. (2016) Interaction networks for learning about objects, relations and physics. Advances in Neural Information Processing Systems29: 6.
7.
BernardiniFMittlemanJRushmeierH, et al. (1999) The ball-pivoting algorithm for surface reconstruction. IEEE Transactions on Visualization and Computer Graphics5(4): 349–359.
8.
ChangPPadırT (2020) Model-based manipulation of linear flexible objects with visual curvature feedback. In: IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), Virtual, July 6-10.IEEE, pp. 1406–1412.
9.
CherubiniAOrtenziVCosgunA, et al. (2020) Model-free vision-based shaping of deformable plastic materials. The International Journal of Robotics Research39(14): 1739–1759.
10.
ChiCBurchfielBCousineauE, et al. (2022) Iterative Residual policy for goal-conditioned dynamic manipulation of deformable objects. In: Proceedings of Robotics: Science and Systems. New York City, NY, USA. DOI: 10.15607/RSS.2022.XVIII.016.
11.
CretuAMPayeurPPetriuEM (2011) Soft object deformation monitoring and learning for model-based robotic hand manipulation. IEEE Transactions on Systems, Man, and Cybernetics - Part B: Cybernetics : A Publication of the IEEE Systems, Man, and Cybernetics Society42(3): 740–753.
12.
CrouseDF (2016) On implementing 2d rectangular assignment algorithms. IEEE Transactions on Aerospace and Electronic Systems52(4): 1679–1696.
13.
EdelsbrunnerHMuckeEP (1994) Three-dimensional alpha shapes. ACM Transactions on Graphics13(1): 43–72.
14.
FanHSuHGuibasLJ (2017) A point set generation network for 3d object reconstruction from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),Hawaii, United States, July 21-26, pp. 605–613.
15.
FelippaCA (2004) Introduction to Finite Element Methods. Boulder, Colorado, United States: University of Colorado, 885.
16.
FigueroaNUrecheALPBillardA (2016) Learning complex sequential tasks from demonstration: a pizza dough rolling case study. In: 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI),Christchurch, NewZealand, March 7-10. IEEE, pp. 611–612.
17.
FletcherR (2013) Practical Methods of Optimization. Hoboken, New Jersey, United States: John Wiley & Sons.
18.
GanapathiASundaresanPThananjeyanB, et al. (2020) Learning to Smooth and Fold Real Fabric Using Dense Object Descriptors Trained on Synthetic Color Images. Cornell University: New York City, NY, UnitedStates. arXiv preprint arXiv:2003.12698.
19.
HaHSongS (2022) Flingbot: the unreasonable effectiveness of dynamic manipulation for cloth unfolding. In: Conference on Robot Learning (CoRL), Auckland, New Zealand, December 14-18. PMLR, pp. 24–33.
20.
HollPThuereyNKoltunV (2019) Learning to control pdes with differentiable physics. In: International Conference on Learning Representations (ICLR), New Orleans, United States, May 6-9.
21.
HoqueRSeitaDBalakrishnaA, et al. (2022) Visuospatial foresight for physical sequential fabric manipulation. Autonomous Robots46(1): 175–199.
22.
HuYAndersonLLiTM, et al. (2019) Difftaichi: differentiable programming for physical simulation. In: International Conference on Learning Representations (ICLR), New Orleans, United States, May 6-9.
23.
HuangZHuYDuT, et al. (2020) Plasticinelab: A soft-body manipulation benchmark with differentiable physics. In: International Conference on Learning Representations (ICLR), Virtual, April 26-May 1.
24.
HuangZLinXHeldD (2022) Mesh-based dynamics model with occlusion reasoning for cloth manipulation. Proceedings of Robotics: Science and Systems (RSS). New York City, NY, United States: RSS.
25.
JatavallabhulaKMMacklinMGolemoF, et al. (2020) Gradsim: Differentiable Simulation for System Identification and Visuomotor Control. In: International Conference on Learning Representations (ICLR), Virtual, April 26-May 1.
26.
KazhdanMBolithoMHoppeH (2006) Poisson surface reconstruction. In: Proceedings of the fourth Eurographics Symposium on Geometry Processing,Goslar, Germany, June26-28. vol. 7.
27.
KingmaDPBaJ (2015) Adam: a method for stochastic optimization. In: International Conference on Learning Representation (ICLR),San Diego, United States, May 7-9.
28.
KurutachTTamarAYangG, et al. (2018) Learning plannable representations with causal infogan. Advances in Neural Information Processing Systems31: 4.
29.
LandauLDLifshitzEMKosevichAM, et al. (1986) Theory of elasticity, vol. 7. London, UK: Elsevier.
30.
Le LidecQKalevatykhILaptevI, et al. (2021) Differentiable simulation for physical system identification. IEEE Robotics and Automation Letters6(2): 3413–3420.
31.
LeeRHamayaMMurookaT, et al. (2021) Sampleefficient learning of deformable linear object manipulation in the real world through self-supervision. IEEE Robotics and Automation Letters7(1): 573–580.
32.
LeggardCMSchranzTSchweigerG, et al. (2021) Constructing neural network-based models for simulating dynamical systems. ACM Computing Surveys1(1): 5.
33.
LiYWuJTedrakeR, et al. (2018) Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. In: International Conference on Learning Representations (ICLR),Vancouver, Canada, April 30-May 3.
34.
LiYLinTYiK, et al. (2020a) Visual grounding of learned physical models. In: International Conference on Machine Learning (ICML), Virtual, July 12-18. PMLR, pp. 5927–5936.
35.
LiYLiSSitzmannV, et al. (2022b) 3d neural scene representations for visuomotor control. In: Conference on Robot Learning (CoRL), Auckland, New Zealand, December 14-18. PMLR, pp. 112–123.
36.
LiYTorralbaAAnandkumarA, et al. (2020b) Causal discovery in physical systems from videos. Advances in Neural Information Processing Systems33: 9180–9192.
37.
LiSHuangZDuT, et al. (2022a) Contact points discovery for soft-body manipulations with differentiable physics. In: International Conference on Learning Representations (ICLR), Virtual, April 25 - 29.
38.
LinXWangYHuangZ, et al. (2022c) Learning visible connectivity dynamics for cloth smoothing. In: Conference on Robot Learning (CoRL), Auckland, New Zealand, December 14-18. PMLR, pp. 256–266.
39.
LinXWangYOlkinJ, et al. (2021) Softgym: Benchmarking deep reinforcement learning for deformable object manipulation. In: Conference on Robot Learning (CoRL), London, UK, November 8-11. PMLR, pp. 432–448.
40.
LinXHuangZLiY, et al. (2022a) Diffskill: skill abstraction from differentiable physics for deformable object manipulations with tools. In: International Conference on Learning Representations (ICLR), Virtual, April 25-29.
41.
LinXQiCZhangY, et al. (2022b) Planning with spatial-temporal abstraction from point clouds for deformable object manipulation. In: 6th Annual Conference on Robot Learning,Auckland, NewZealand, December 14-18.
42.
LuoYXuHLiY, et al. (2018) Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees. In: International Conference on Learning Representations (ICLR),Vancouver, Canada, April 30-May 3.
43.
ManuelliLLiYFlorenceP, et al. (2021) Keypoints into the future: self-supervised correspondence in model-based reinforcement learning. In: Conference on Robot Learning (CoRL), London, UK, November 8-11. PMLR, pp. 693–710.
44.
MatasJJamesSDavisonAJ (2018) Sim-to-real reinforcement learning for deformable object manipulation. In: Conference on Robot Learning (CoRL), Zürich, Switzerland, October 29-31. PMLR, pp. 734–743.
45.
MatlCBajcsyR (2021) Deformable elasto-plastic object shaping using an elastic hand and model-based reinforcement learning. In: 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Prague, Czech Republic, September 27-October 1.IEEE, pp. 3955–3962.
46.
MillerSVan Den BergJFritzM, et al. (2012) A geometric approach to robotic laundry folding. The International Journal of Robotics Research31(2): 249–267.
47.
MonaghanJJ (1992) Smoothed particle hydrodynamics. Annual Review of Astronomy and Astrophysics30(1): 543–574.
48.
MrowcaDZhuangCWangE, et al. (2018) Flexible neural representation for physics prediction. Advances in Neural Information Processing Systems31: 1.
49.
MullerMHeidelbergerBHennixM, et al. (2007) Position based dynamics. Journal of Visual Communication and Image Representation18(2): 109–118.
50.
NadonFValenciaAJPayeurP (2018) Multi-modal sensing and robotic manipulation of non-rigid objects: a survey. Robotics7(4): 74.
51.
NagabandiAKonoligeKLevineS, et al. (2020) Deep dynamics models for learning dexterous manipulation. In: Conference on Robot Learning (CoRL), Virtual, November 16-18. PMLR, pp. 1101–1112.
52.
NairAChenDAgrawalP, et al. (2017) Combining self-supervised learning and imitation for vision-based rope manipulation. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), Marina Bay Sands, Singapore, May 29-June 3.IEEE, pp. 2146–2153.
53.
Navarro-AlarconDYipHMWangZ, et al. (2016) Automatic 3-d manipulation of soft objects by robotic arms with an adaptive deformation model. IEEE Transactions on Robotics32(2): 429–441.
54.
ReddyJN (2019) Introduction to the Finite Element Method. New York City, NY, United States: McGraw-Hill Education.
55.
RubnerYTomasiCGuibasLJ (1998) A metric for distributions with applications to image databases. In: Sixth International Conference on Computer Vision (IEEE Cat. No. 98CH36271),Bombay, India, January 7. IEEE, pp. 59–66.
56.
SanchezJCorralesJABouzgarrouBC, et al. (2018) Robotic manipulation and sensing of deformable objects in domestic and industrial applications: a survey. The International Journal of Robotics Research37(7): 688–716.
57.
Sanchez-GonzalezAGodwinJPfaffT, et al. (2020) Learning to simulate complex physics with graph networks. In: International Conference on Machine Learning (ICML), Virtual, July 12-18. PMLR, pp. 8459–8468.
58.
SanchezDWanWHaradaK (2020) Tethered tool manipulation planning with cable maneuvering. IEEE Robotics and Automation Letters5(2): 2777–2784.
59.
SchoenholzSSCubukEDJaxM (2020) End-to-end differentiable, hardware accelerated, molecular dynamics in pure python. Bulletin of the American Physical Society65: APS.
60.
SchrittwieserJAntonoglouIHubertT, et al. (2020) Mastering atari, go, chess and shogi by planning with a learned model. Nature588(7839): 604–609.
61.
SeitaDFlorencePTompsonJ, et al. (2021) Learning to rearrange deformable cables, fabrics, and bags with goal-conditioned transporter networks. In: 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi’an, China, May 30 - June 5. IEEE, pp. 4568–4575.
62.
SheYWangSDongS, et al. (2021) Cable manipulation with a tactile-reactive gripper. The International Journal of Robotics Research40(14): 1385–1401.
63.
ShlomiJBattagliaPVlimantJR (2020) Graph neural networks in particle physics. Machine Learning: Science and Technology2(2): 021001.
64.
SulskyDZhouSJSchreyerHL (1995) Application of a particle-in-cell method to solid mechanics. Computer Physics Communications87(1-2): 236–252.
65.
SundaresanPGrannenJThananjeyanB, et al. (2020) Learning rope manipulation policies using dense object descriptors trained on synthetic depth data. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), Virtual, May 31 - June15.IEEE, pp. 9411–9418.
66.
ThananjeyanBGargAKrishnanS, et al. (2017) Multilateral surgical pattern cutting in 2d orthotropic gauze with deep reinforcement learning policies for tensioning. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), Marina Bay Sands, Singapore, May 29-June 3.IEEE, pp. 2371–2378.
67.
VictoriaAHMaragathamG (2021) Automatic tuning of hyperparameters using bayesian optimization. Evolving Systems12: 217–223.
68.
ViswanathVShivakumarKKerrJ, et al. (2022) Autonomously Untangling long cables. In: Proceedings of Robotics: Science and Systems. New York City, NY, USA. DOI: 10.15607/RSS.2022.XVIII.034.
69.
WangAKurutachTLiuK, et al. (2019) Learning robotic manipulation through visual planning and acting. Proceedings of Robotics: Science and Systems (RSS). Freiburg, Germany: RSS.
70.
WuYYanWKurutachT, et al. (2020) Learning to manipulate deformable objects without demonstrations. Proceedings of Robotics: Science and Systems (RSS). Virtual: RSS.
71.
XuZChiCBurchfielB, et al. (2022) DextAIRity: deformable manipulation can be a Breeze. In: Proceedings of Robotics: Science and Systems. New York City, NY, USA. DOI: 10.15607/RSS.2022.XVIII.017.
72.
YanMZhuYJinN, et al. (2020) Self-supervised learning of state estimation for manipulating deformable linear objects. IEEE Robotics and Automation Letters5(2): 2372–2379.
73.
YanWVangipuramAAbbeelP, et al. (2021) Learning predictive representations for deformable objects using contrastive estimation. In: Conference on Robot Learning (CoRL), London, UK, November 8-11. PMLR, pp. 564–574.
74.
YinHVaravaAKragicD (2021) Modeling, learning, perception, and control methods for deformable object manipulation. Science Robotics6(54): 2.
75.
YoshimotoKHigashimoriMTadakumaK, et al. (2011) Active outline shaping of a rheological object based on plastic deformation distribution. In: 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems,San Francisco, United States, September 25-30. IEEE, pp. 1386–1391.
76.
YukselC (2015) Sample elimination for generating Poisson disk sample sets. Computer Graphics Forum34: 25–32. Wiley Online Library.
77.
ZimmermanWB (2006) Multiphysics modeling with finite element methods. World Scientific Publishing Company18: 45.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
