In this work, we propose a vision-based and finite element method (FEM)-based controller to automate the 3D shaping of soft objects with dual-arm robots. Our controller relies on a data-based approach to learn how the robot’s actions result in object deformations, while also running FEM-based simulations to infer the shape of the whole body. These model-based simulations are used to generate initial shape data, allowing to extract visual features through a principal component analysis and thus estimate the interaction matrix of the object–robot system. In contrast with most existing shape servoing controllers, our new model-based approach continuously predicts the object deformations produced by the robot, which are then compared to the visually observed deformation feedback. This iterative process enables to correct the deformed mesh model before updating the interaction matrix. To validate this new control methodology, we present a detailed experimental study with a dual-arm robot and different soft objects, which showcases the performance of our automatic shaping framework.
AghajanzadehOArandaMLópez-NicolásG, et al. (2022a) An offline geometric model for controlling the shape of elastic linear objects. In: 2022 IEEE/RSJ international conference on intelligent robots and systems (IROS), Kyoto, Japan, 23–27 October 2022, pp. 2175–2181. DOI: 10.1109/IROS47612.2022.9981466.
2.
AghajanzadehOPicardGRamonJAC, et al. (2022b) Optimal deformation control framework for elastic linear objects. In: 2022 IEEE 18th international conference on automation science and engineering (CASE), Mexico City, Mexico, 20–24 August 2022, pp. 722–728. DOI: 10.1109/CASE49997.2022.9926528.
3.
AndronasDKampourakisEBakopoulouK, et al. (2021) Model-based robot control for human-robot flexible material co-manipulation. In: 2021 26th IEEE international conference on emerging technologies and factory automation (ETFA), Vasteras, Sweden, 7–10 September 2021, pp. 1–8. DOI: 10.1109/ETFA45728.2021.9613235.
4.
ArmenàkasAE (2016) Advanced Mechanics of Materials and Applied Elasticity. Boca Raton, FL: CRC Press.
5.
AzadAQuentinHFaizBA, et al. (2023) Optimal cosserat-based deformation control for robotic manipulation of linear objects. In: 2023 IEEE/ASME international conference on advanced intelligent mechatronics (AIM), Seattle, WA, 28–30 June 2023, pp. 381–388. DOI: 10.1109/AIM46323.2023.10196216.
6.
CaporaliAZanellaRGreogrioDD, et al. (2022) Ariadne+: deep learning–based augmented framework for the instance segmentation of wires. IEEE Transactions on Industrial Informatics18(12): 8607–8617. DOI: 10.1109/TII.2022.3154477.
7.
ChaumetteF (2007) Visual Servoing, Chapter 6. Hoboken, NJ: John Wiley and Sons, Ltd, 279–336. DOI: 10.1002/9780470612286.ch6.
8.
CoevoetEMorales-BiezeTLargilliereF, et al. (2017) Software toolkit for modeling, simulation, and control of soft robots. Advanced Robotics31(22): 1208–1224.
9.
FaureFDuriezCDelingetteH, et al. (2012) Sofa: a multi-model framework for interactive physical simulation. In: Soft Tissue Biomechanical Modeling for Computer Assisted Surgery. Berlin, Germany: Springer, 283–321.
10.
FicucielloFMigliozziACoevoetE, et al. (2018) Fem-based deformation control for dexterous manipulation of 3d soft objects. In: 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS), Madrid, Spain, 1–5 October 2018, 4007–4013. IEEE.
11.
GouryODuriezC (2018) Fast, generic, and reliable control and simulation of soft robots using model order reduction. IEEE Transactions on Robotics34(6): 1565–1576. DOI: 10.1109/TRO.2018.2861900.
12.
HuZHanTSunP, et al. (2019) 3-d deformable object manipulation using deep neural networks. IEEE Robotics and Automation Letters4(4): 4255–4261. DOI: 10.1109/LRA.2019.2930476.
13.
LagneauRKrupaAMarchalM (2020) Automatic shape control of deformable wires based on model-free visual servoing. IEEE Robotics and Automation Letters5(4): 5252–5259.
14.
LeeRHamayaMMurookaT, et al. (2022) Sample-efficient learning of deformable linear object manipulation in the real world through self-supervision. IEEE Robotics and Automation Letters7(1): 573–580. DOI: 10.1109/LRA.2021.3130377.
15.
MaZXiaoJ (2023) Robotic perception-motion synergy for novel rope wrapping tasks. IEEE Robotics and Automation Letters8(7): 4131–4138.
16.
MacQueenJ (1967) Classification and analysis of multivariate observations. In: Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability. Los Angeles, CA: University of California Los Angeles, 281–297.
17.
MakiyehFChaumetteFMarchalM, et al. (2023) Shape servoing of a soft object using fourier series and a physics-based model. In: IEEE/RSJ international conference on intelligent robots and systems, Detroit, MI, 1–5 October 2023, pp. 6356–6363.
18.
ManeewongvatanaSMountDM (1999) Analysis of approximate nearest neighbor searching with clustered point sets. ArXiv Preprint cs/9901013.
19.
Navarro-AlarconDYipHMWangZ, et al. (2016) Automatic 3-d manipulation of soft objects by robotic arms with an adaptive deformation model. IEEE Transactions on Robotics32(2): 429–441. DOI: 10.1109/TRO.2016.2533639.
20.
QiJMaGZhuJ, et al. (2022) Contour moments based manipulation of composite rigid-deformable objects with finite time model estimation and shape/position control. IEEE27(5): 2985–2996. DOI: 10.1109/TMECH.2021.3126383.
21.
QiJZhouPRanG, et al. (2024) Model predictive manipulation of compliant objects with multi-objective optimizer and adversarial network for occlusion compensation. ISA Transactions150: 359–373. DOI: 10.1016/j.isatra.2024.05.015.
22.
RuanMMcConachieDBerensonD (2018) Accounting for directional rigidity and constraints in control for manipulation of deformable objects without physical simulation. In: 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS), Madrid, Spain, 1–5 October 2018, pp. 512–519. DOI: 10.1109/IROS.2018.8594520.
23.
SaghourCCélérierMFraisseP, et al. (2023) Visual servoing for dual arm shaping of soft objects in 3d. In: IEEE-RAS 22nd international conference on humanoid robots (humanoids), Austin, TX, 12–14 December 2023.
24.
SeitaDFlorencePTompsonJ, et al. (2021) Learning to rearrange deformable cables, fabrics, and bags with goal-conditioned transporter networks. In: 2021 IEEE international conference on robotics and automation (ICRA), Xi’an, China, 30 May 2021–05 June 2021, pp. 4568–4575. DOI: 10.1109/ICRA48506.2021.9561391.
25.
Shetab-BushehriMArandaMMezouarY, et al. (2022) Lattice-based shape tracking and servoing of elastic objects. ArXiv preprint arXiv:2209.01832.
26.
ShinCFergusonPWPedramSA, et al. (2019) Autonomous tissue manipulation via surgical robot using learning based model predictive control. In: 2019 international conference on robotics and automation (ICRA), Montreal, QC, 20–24 May 2019, 3875–3881. IEEE.
27.
TsurumineYCuiYUchibeE, et al. (2019) Deep reinforcement learning with smooth policy update: application to robotic cloth manipulation. Robotics and Autonomous Systems112: 72–83. DOI: 10.1016/j.robot.2018.11.004.
28.
WangTYamakawaY (2023) Edge-supervised linear object skeletonization for high-speed camera. Sensors23(12): 5721. DOI: 10.3390/s23125721.
29.
YuMLvKWangC, et al. (2023) Generalizable whole-body global manipulation of deformable linear objects by dual-arm robot in 3-d constrained environments. ArXiv preprint arXiv:2310.09899.
30.
ZhouPZhengPQiJ, et al. (2024) Reactive human–robot collaborative manipulation of deformable linear objects using a new topological latent control model. Robotics and Computer-Integrated Manufacturing88: 102727.
31.
ZhuJNavarro-AlarconDPassamaR, et al. (2020) Vision-based manipulation of deformable and rigid objects using subspace projections of 2d contours. Robotics and Autonomous Systems142: 103798.
32.
ZhuJCherubiniADuneC, et al. (2022) Challenges and outlook in robotic manipulation of deformable objects. IEEE Robotics and Automation Magazine29(3): 67–77. DOI: 10.1109/MRA.2022.3147415.
33.
ZimmermannSPoranneRCorosS (2021) Dynamic manipulation of deformable objects with implicit integration. IEEE Robotics and Automation Letters6(2): 4209–4216. DOI: 10.1109/LRA.2021.3066969.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
