Self-modeling enables robots to learn task-agnostic models of their morphology and kinematics based on data that can be automatically collected, with minimal human intervention and prior information, thereby enhancing machine intelligence. Recent research has highlighted the potential of data-driven technology in modeling the morphology and kinematics of robots. However, existing self-modeling methods suffer from either low modeling quality or excessive data acquisition costs in equipment. Beyond morphology and kinematics, surface color is also a crucial component of robots, which is challenging to model and remains unexplored. In this work, a high-quality, surface color-aware, and link-level method is proposed for robot self-modeling. We utilize three-dimensional (3D) Gaussians to represent the static morphology and surface color of robots, and cluster the 3D Gaussians to construct neural ellipsoid bones, whose deformations are controlled by the transformation matrices generated by a kinematic neural network. The 3D Gaussians and kinematic neural network are trained using data pairs composed of joint angles, camera parameters, and multi-view images without depth information. By feeding the kinematic neural network with joint angles, we can utilize the well-trained model to describe the corresponding morphology, kinematics, and surface color of robots at the link level, and render robot images from different perspectives with the aid of 3D Gaussian splatting. Furthermore, we demonstrate that the established model can be exploited to perform downstream tasks such as motion planning and inverse kinematics.
ArthurDVassilvitskiiS (2007) k-means++: the advantages of careful seeding. In: Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA ’07. USA. Society for Industrial and Applied Mathematics, 1027–1035.
2.
BongardJZykovVLipsonH (2006) Resilient machines through continuous self-modeling. Science314(5802): 1118–1121.
3.
ChenBHuYLiL, et al. (2021) Smile like you mean it: driving animatronic robotic face with learned models. In: 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2739–2746.
4.
ChenBKwiatkowskiRVondrickC, et al. (2022) Fully body visual self-modeling of robot morphologies. Science Robotics7(68): eabn1944.
5.
ColemanDSucanIChittaS, et al. (2014) Reducing the barrier to entry of complex robotic software: a moveit! case study. arXiv preprint arXiv:1404.3785.
6.
CoumansEBaiY (2016–2021) Pybullet, a python module for physics simulation for games, Robotics andC Learning. https://pybullet.org/
7.
Díaz LedezmaFHaddadinS (2023) Machine learning–driven self-discovery of the robot body morphology. Science Robotics8(85): eadh0972.
8.
EdelsbrunnerHKirkpatrickDSeidelR (1983) On the shape of a set of points in the plane. IEEE Transactions on Information Theory29(4): 551–559.
9.
FangJYiTWangX, et al. (2022) Fast dynamic radiance fields with time-aware neural voxels. In: SIGGRAPH Asia 2022 Conference Papers, Daegu, Republic of Korea, December 6-9, 2022, 1–9.
10.
HangKBircherWGMorganAS, et al. (2021) Manipulation for self-identification, and self-identification for better manipulation. Science Robotics6(54): eabe1321.
11.
HoreAZiouD (2010) Image quality metrics: PSNR vs. SSIM. In: 2010 20th International Conference on Pattern Recognition. IEEE, 2366–2369.
12.
HuYLinJLipsonH (2023) Teaching robots to build simulations of themselves. arXiv preprint arXiv:2311.12151.
13.
HuSHuTLiuZ (2024) Gauhuman: articulated gaussian splatting from monocular human videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 20418–20431.
14.
HuangYHSunYTYangZ, et al. (2024) Sc-gs: sparse-controlled gaussian splatting for editable dynamic scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4220–4230.
15.
JohnsonJRaviNReizensteinJ, et al. (2020) Accelerating 3D deep learning with PyTorch3D. In: SIGGRAPH Asia 2020 Courses, SA ’20: Association for Computing Machinery.
KirillovAMintunERaviN, et al. (2023) Segment anything. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. IEEE, 4015–4026.
19.
KoenigNHowardA (2004) Design and use paradigms for gazebo, an open-source multi-robot simulator. In: 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, Vol. 3, 2149–2154.
20.
KwiatkowskiR (2022) Deep Self-Modeling for Robotic Systems. PhD Thesis. Columbia University.
LabbéYCarpentierJAubryM, et al. (2021) Single-view robot pose and joint angle estimation via render & compare. In:Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 1654–1663.
23.
LiangJMakoviychukVHandaA, et al. (2018) GPU-Accelerated robotic simulation for distributed reinforcement learning. In: Conference on Robot Learning. PMLR, 270–282.
24.
LiangYKhanNLiZ, et al. (2025) Gaufre: gaussian deformation fields for real-time dynamic novel view synthesis. In: 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE, 2642–2652.
25.
LinYDaiZZhuS, et al. (2024) Gaussian-flow: 4D reconstruction with dynamic 3D gaussian particle. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 21136–21145.
LouHLiuYPanY, et al. (2024) Robo-GS: a physics consistent spatial-temporal model for robotic arm with hybrid representation. https://arxiv.org/abs/2408.14873
31.
LuJLiuFGirerdC, et al. (2023) Image-based pose estimation and shape reconstruction for robot manipulators and soft, continuum robots via differentiable rendering. In: 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 560–567.
32.
LuitenJKopanasGLeibeB, et al. (2024) Dynamic 3D gaussians: tracking by persistent dynamic view synthesis2024 International Conference on 3D Vision (3DV). IEEE, 800–809.
33.
Marques MonteiroRShiJWurdemannH, et al. (2024) Visuo-dynamic self-modelling of soft robotic systems. Frontiers in Robotics and AI11: 1403733.
34.
MathewMSapraRMajumderS (2014) A learning based approach to self modeling robots. In: 2014 International Conference on Control, Instrumentation, Communication and Computational Technologies (ICCICCT). IEEE, 758–762.
35.
MildenhallBSrinivasanPPTancikM, et al. (2021) Nerf: representing scenes as neural radiance fields for view synthesis. Communications of the ACM65(1): 99–106.
36.
MüllerTEvansASchiedC, et al. (2022) Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics41(4): 102:1–102:15.
37.
NoguchiASunXLinS, et al. (2021) Neural articulated radiance field. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 5762–5772.
38.
NoguchiAIqbalUTremblayJ, et al. (2022) Watch it move: unsupervised discovery of 3D joints for re-posing of articulated objects. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 3677–3687.
39.
OdenaAOlahCShlensJ (2017) Conditional image synthesis with auxiliary classifier gans. In: International Conference on Machine Learning. PMLR, 2642–2651.
40.
PedregosaFVaroquauxGGramfortA, et al. (2011) Scikit-learn: machine learning in python. Journal of Machine Learning Research12: 2825–2830.
41.
ProkudinSMaQRaafatM, et al. (2023) Dynamic point fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 7964–7976.
42.
PumarolaACoronaEPons-MollG, et al. (2021) D-NeRF: neural radiance fields for dynamic scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 10318–10327.
43.
QianZWangSMihajlovicM, et al. (2024) 3DGS-avatar: animatable avatars via deformable 3D gaussian splatting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 5020–5030.
44.
RochatP (2003) Five levels of self-awareness as they unfold early in life. Consciousness and Cognition12(4): 717–731.
45.
Sanchez-GonzalezAHeessNSpringenbergJT, et al. (2018) Graph networks as learnable physics engines for inference and control. In: International Conference on Machine Learning. PMLR, 4470–4479.
46.
SchönbergerJLFrahmJM (2016) Structure-from-motion revisited. In: Conference on Computer Vision and Pattern Recognition (CVPR). IEEE.
47.
SchulzeLLipsonH (2024) High-degrees-of-freedom dynamic neural fields for robot self-modeling and motion planning. In: 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 3064–3070.
TertikasKPaschalidouDPanB, et al. (2023) Generating part-aware editable 3D shapes without 3D supervision. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 4466–4478.
50.
TodorovEErezTTassaY (2012) Mujoco: a physics engine for model-based control. In: 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 5026–5033.
51.
UzolasLEisemannEKellnhoferP (2023) Template-free articulated neural point clouds for reposable view synthesis. Advances in Neural Information Processing Systems36: 31621–31637.
52.
WangZBovikACSheikhHR, et al. (2004) Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing: A Publication of the IEEE Signal Processing Society13(4): 600–612.
53.
WuGYiTFangJ, et al. (2024) 4D gaussian splatting for real-time dynamic scene rendering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 20310–20320.
54.
YangGSunDJampaniV, et al. (2021) Lasr: learning articulated shape reconstruction from a monocular video. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 15980–15989.
55.
YangGVoMNeverovaN, et al. (2022) Banmo: building animatable 3D neural models from many casual videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2863–2873.
56.
YangYLiuJZhouH, et al. (2024a) Robotsdf: implicit morphology modeling for the robotic arm. Sensors24(16): 5248.
57.
YangZGaoXZhouW, et al. (2024b) Deformable 3D gaussians for high-fidelity monocular dynamic scene reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 20331–20341.
58.
YangZYangHPanZ, et al. (2024c) Real-time photorealistic dynamic scene representation and rendering with 4D Gaussian splatting. In: International Conference on Learning Representations (ICLR), Singapore, 24-28, April 2025.
59.
YuHJulinJMilacskiZÁ, et al. (2024) Cogs: controllable Gaussian splatting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 21624–21633.
60.
ZhangRIsolaPEfrosAA, et al. (2018) The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 586–595.
61.
ZhangTGaoQLiW, et al. (2024) Bags: building animatable Gaussian splatting from a monocular video with diffusion priors. https://arxiv.org/abs/2403.11427
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
