How can we build general-purpose robot systems for open-world semantic navigation, for example, searching a novel environment for a target object specified in natural language? To tackle this challenge, we introduce OSG Navigator, a modular system composed of foundation models, for open-world Object-Goal Navigation (ObjectNav). Foundation models provide enormous semantic knowledge about the world, but struggle to organise and maintain spatial information effectively at scale. Key to OSG Navigator is the Open Scene Graph representation, which acts as spatial memory for OSG Navigator. It organises spatial information hierarchically using OSG schemas, which are templates, each describing the common structure of a class of environments. OSG schemas can be automatically generated from simple semantic labels of a given environment, for example, ‘home’ or ‘supermarket’. They enable OSG Navigator to adapt zero-shot to new environment types. We conducted experiments using both Fetch and Spot robots in simulation and in the real world, showing that OSG Navigator achieves state-of-the-art performance on ObjectNav benchmarks and generalises zero-shot over diverse goals, environments and robot embodiments.
Al-HalahZRamakrishnanSKGraumanK (2022) Zero experience required: plug & play modular transfer learning for semantic visual navigation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), New Orleans, LA, USA, 18–24 June 2022, 17031–17041.
2.
AmiriSChandanKZhangS (2022) Reasoning with scene graphs for robot planning under partial observability. IEEE Robotics and Automation Letters7(2): 5560–5567.
3.
AndersonPChangAChaplotDS, et al. (2018) On evaluation of embodied navigation agents. arXiv.
4.
AnwarAWelshJBiswasJ, et al. (2024) Remembr: building and reasoning over long-horizon spatio-temporal memory for robot navigation. arXiv.
5.
ArmeniIHeZYGwakJ, et al. (2019) 3D scene graph: a structure for unified semantics, 3D space, and camera. In: 2019 IEEE/CVF international conference on computer vision (ICCV), Seoul, South Korea, 27 October - 2 November 2019, 5663–5672.
6.
BrownTMannBRyderN, et al. (2020) Language models are few-shot learners. In: LarochelleHRanzatoMHadsellR, et al. (eds) Advances in Neural Information Processing Systems. Curran Associates, Inc., Vol. 33, 1877–1901.
7.
CaiWHuangSChengG, et al. (2023) Bridging zero-shot object navigation and foundation models through pixel-guided navigation skill. arXiv.
8.
ChangMGervetTKhannaM, et al. (2023) Goat: go to any thing. arXiv.
9.
ChaplotDSGandhiDPGuptaA, et al. (2020) Object goal navigation using goal-oriented semantic exploration. In: LarochelleHRanzatoMHadsellR, et al. (eds) Advances in Neural Information Processing Systems. Curran Associates, Inc., Vol. 33, 4247–4258.
10.
ChenJLiGKumarS, et al. (2023) How to not train your dragon: training-free embodied object goal navigation with semantic frontiers. In: Proceedings of robotics: science and systems, Daegu, Republic of Korea, 10–14 July 2023.
11.
ChenJLinBXuR, et al. (2024) Mapgpt: map-guided prompting for unified vision-and-language navigation. arXiv.
12.
DaiZAsgharivaskasiADuongT, et al. (2024) Optimal scene graph planning with large language model guidance. In: 2024 IEEE international conference on robotics and automation (ICRA),Yokohama, Japan, 13–17 May 2024, 14062–14069.
13.
DeitkeMClarkCLeeS, et al. (2024) Molmo and pixmo: open weights and open data for state-of-the-art multimodal models. arXiv.
14.
DorbalaVSMullenJFManochaD (2024) Can an embodied agent find your “cat-shaped mug”? LLM-based zero-shot object navigation. IEEE Robotics and Automation Letters9(5): 4083–4090.
15.
DubeyAJauhriAPandeyA, et al. (2024) The llama 3 herd of models. arXiv.
16.
GadreSYWortsmanMIlharcoG, et al. (2023) Cows on pasture: baselines and benchmarks for language-driven zero-shot object navigation. In: CVPR, Vancouver, 18–22 June 2023.
17.
GrinvaldMFurrerFNovkovicT, et al. (2019) Volumetric instance-aware semantic mapping and 3D object discovery. IEEE Robotics and Automation Letters4(3): 3037–3044.
18.
GuJStefaniEWuQ, et al. (2022) Vision-and-language navigation: a survey of tasks, methods, and future directions. In: Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), Dublin, Ireland, 22–27 May 2022, 7606–7623.
19.
GuQKuwajerwalaAMorinS, et al. (2023) Conceptgraphs: open-vocabulary 3D scene graphs for perception and planning. arXiv.
20.
GuptaADollarPGirshickR (2019) LVIS: a dataset for large vocabulary instance segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Long Beach, CA, USA, 15–20 June 15 2019, 5351–5359.
21.
HonerkampDBüchnerMDespinoyF, et al. (2024) Language-grounded dynamic scene graphs for interactive object search with mobile manipulation. arXiv preprint arXiv:2403.08605.
22.
HuYXieQJainV, et al. (2023) Toward general-purpose robots via foundation models: a survey and meta-analysis. arXiv.
23.
HuangJChangKCC (2022) Towards reasoning in large language models: a survey. arXiv preprint arXiv:2212.10403.
24.
HuangCZhangL (2025) On the limit of language models as planning formalizers. arXiv.
25.
HuangWAbbeelPPathakD, et al. (2022) Language models as zero-shot planners: extracting actionable knowledge for embodied agents. In: ChaudhuriKJegelkaSSongL, et al. (eds) Proceedings of the 39th international conference on machine learning, proceedings of machine learning research, Baltimore, MA, 17–23 July 2022. PMLR, Vol. 162, 9118–9147.
26.
HughesNChangYCarloneL (2022) Hydra: a real-time spatial perception system for 3D scene graph construction and optimization. In: Proceedings of robotics: science and systems. New York City, NY, USA, 2022.
27.
HutchinsonSHagerGCorkeP (1996) A tutorial on visual servo control. IEEE Transactions on Robotics and Automation12: 651–670.
28.
IchterBBrohanAChebotarY, et al. (2023) Do as i can, not as i say: grounding language in robotic affordances. In: LiuKKulicDIchnowskiJ (eds) Proceedings of the 6th conference on robot learning, proceedings of machine learning research, Auckland, New Zealand, 14–18 December 2022, PMLR, Vol. 205, 287–318.
29.
JatavallabhulaKKuwajerwalaAGuQ, et al. (2023) Conceptfusion: open-set multimodal 3D mapping. Robotics: Science and Systems (RSS)XIX: 066.
30.
KassabCMattamalaMZhangL, et al. (2024) Language-extended indoor slam (lexis): a versatile system for real-time visual scene understanding. In: 2024 IEEE international conference on robotics and automation (ICRA),Yokohama, Japan, 13–17 May 2024, 15988–15994.
31.
KhandelwalAWeihsLMottaghiR, et al. (2022) Simple but effective: clip embeddings for embodied AI. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), New Orleans, LA, USA, 18–24 June 2022, 14829–14838.
32.
KirillovAMintunERaviN, et al. (2023) Segment anything. arXiv:2304.02643.
33.
KrantzJLeeSMalikJ, et al. (2022) Instance-specific image goal navigation: training embodied agents to find object instances.
34.
KrantzJBanerjeeSZhuW, et al. (2023) Iterative vision-and-language navigation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), Vancouver, BC, Canada, 17–24 June 2023, 14921–14930.
35.
LewisPPerezEPiktusA, et al. (2020) Retrieval-augmented generation for knowledge-intensive nlp tasks. In: LarochelleHRanzatoMHadsellR, et al. (eds) Advances in Neural Information Processing Systems. Curran Associates, Inc., Vol. 33, 9459–9474.
36.
LiJLiDSavareseS, et al. (2023) BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In: KrauseABrunskillEChoK, et al. (eds) Proceedings of the 40th International conference on machine learning, proceedings of machine learning research, Honolulu, Hawaii, 23–29 July 2023. PMLR, Vol. 202, 19730–19742.
37.
LiuJShenDZhangY, et al. (2022) What makes good in-context examples for GPT-3? In: AgirreEApidianakiMVulićI (eds) Proceedings of deep learning inside out (DeeLIO 2022): The 3rd workshop on knowledge extraction and integration for deep learning architectures. Dublin, Ireland and Online. Association for Computational Linguistics,100–114.
38.
LiuHLiCWuQ, et al. (2023a) Visual instruction tuning.
39.
LiuSZengZRenT, et al. (2023b) Grounding DINO: marrying DINO with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499.
40.
LongYCaiWWangH, et al. (2025) Instructnav: zero-shot system for generic instruction navigation in unexplored environment. In: AgrawalPKroemerOBurgardW (eds) Proceedings of the 8th conference on robot learning, proceedings of machine learning research,Munich, Germany, 6–9 Nov 2024. PMLR, Vol. 270, 2049–2060.
41.
MaggioDChangYHughesN, et al. (2024) Clio: real-time task-driven open-set 3D scene graphs. IEEE Robotics and Automation Letters9(10): 8921–8928.
42.
MajumdarAAggarwalGDevnaniB, et al. (2022) Zson: zero-shot object-goal navigation using multimodal goal embeddings. In: KoyejoSMohamedSAgarwalA, et al. (eds) Advances in Neural Information Processing Systems. Curran Associates, Inc., Vol. 35, 32340–32352.
43.
MajumdarAXiaFIchterB, et al. (2023) Findthis: language-driven object disambiguation in indoor environments. In: TanJToussaintMDarvishK (eds) Proceedings of the 7th conference on robot learning, proceedings of machine learning research,Atlanta, 6–9 Nov2023. PMLR, Vol. volume 229, 1335–1347.
44.
MaksymetsOCartillierVGokaslanA, et al. (2021) THDA: treasure hunt data augmentation for semantic navigation. In: Proceedings of the IEEE/CVF international conference on computer vision (ICCV), Montreal, QC, Canada, 11–17 October 2021, 15374–15383.
45.
MascaroRChliM (2024) Scene representations for robotic spatial perception. Annual Review of Control, Robotics, and Autonomous Systems8: 351–377.
46.
MindererMGritsenkoAStoneA, et al. (2022) Simple open-vocabulary object detection. In: Computer vision – ECCV 2022: 17th European conference, Tel Aviv, Israel, 23–27 October 2022, Proceedings, Part X. Springer-Verlag, 728–755.
47.
MousavianAToshevAFišerM, et al. (2019) Visual representations for semantic target driven navigation. In: 2019 international conference on robotics and automation (ICRA), Montreal, Quebec, Canada, 20–24 May 2019. IEEE Press, 8846–8852.
48.
AchiamJAdlerSAgarwalSOpenAI, et al (2024) Gpt-4 technical report. arXiv.
49.
OquabMDarcetTMoutakanniT, et al. (2023) Dinov2: learning robust visual features without supervision. arXiv.
50.
PengSGenovaKJiangC, et al. (2023) Openscene: 3D scene understanding with open vocabularies. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), Vancouver, 18–22 June 2023, 815–824.
51.
QinMLiWZhouJ, et al. (2024) LangSplat: 3D language Gaussian splatting. In: 2024 IEEE/CVF conference on computer vision and pattern recognition (CVPR), Seattle, WA, USA, 1-18 June 2024, IEEE Computer Society, 20051–20060.
52.
RadfordAKimJWHallacyC, et al. (2021) Learning transferable visual models from natural language supervision. In: MeilaMZhangT (eds) Proceedings of the 38th international conference on machine learning, proceedings of machine learning research, Virtual, 18–24 July2021. PMLR, Vol. 139, 8748–8763.
53.
RajvanshiASikkaKLinX, et al. (2023) Saynav: grounding large language models for dynamic planning to navigation in new environments. arXiv.
54.
RamakrishnanSKGokaslanAWijmansE, et al. (2021) Habitat-matterport 3D dataset (HM3D): 1000 large-scale 3D environments for embodied AI. In: Thirty-fifth conference on neural information processing systems datasets and benchmarks track, Virtual, 6–14Dec 2021.
55.
RamakrishnanSKChaplotDSAl-HalahZ, et al. (2022) Poni: potential functions for objectgoal navigation with interaction-free learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR),New Orleans, Louisiana, USA, 19–24 June 2022, 18890–18900.
56.
RanaKHavilandJGargS, et al. (2023) Sayplan: grounding large language models using 3D scene graphs for scalable robot task planning. In: TanJToussaintMDarvishK (eds) Proceedings of the 7th conference on robot learning, proceedings of machine learning research, Atlanta, USA, 19–23 May 2025. PMLR, Vol. volume 229, 23–72.
57.
RanganathanADellaertF (2011) Online probabilistic topological mapping. The International Journal of Robotics Research30(6): 755–771.
58.
RavichandranZPengLHughesN, et al. (2022) Hierarchical representations and explicit memory: learning effective navigation policies on 3D scene graphs using graph neural networks. In: 2022 international conference on robotics and automation (ICRA), Philadelphia, PA, 23–27 May 2022, 9272–9279.
59.
RavichandranZMuraliVTzesM, et al. (2025) Spine: online semantic planning for missions with incomplete natural language specifications in unstructured environments. In: International conference on robotics and automation (ICRA), Atlanta, USA, 19–23 May 2025.
60.
RayABradleyCCarloneL, et al. (2024) Task and motion planning in hierarchical 3D scene graphs. arXiv.
61.
RenAZClarkJDixitA, et al. (2024) Explore until confident: efficient exploration for embodied question answering. arXiv.
62.
RosinolAVioletteAAbateM, et al. (2021) Kimera: from slam to spatial perception with 3D dynamic scene graphs. The International Journal of Robotics Research40(12-14): 1510–1546.
63.
SethianJA (1996) A fast marching level set method for monotonically advancing fronts. Proceedings of the National Academy of Sciences of the United States of America93(4): 1591–1595.
64.
ShahDEquiMROsińskiB, et al. (2023a) Navigation with large language models: semantic guesswork as a heuristic for planning. In: TanJToussaintMDarvishK (eds) Proceedings of the 7th conference on robot learning, proceedings of machine learning research, Atlanta, GA, USA, 6–9 November 2023. PMLR, Vol. 229, 2683–2699.
65.
ShahDSridharADashoraN, et al. (2023b) Vint: a foundation model for visual navigation. In: TanJToussaintMDarvishK (eds) Proceedings of the 7th conference on robot learning, proceedings of machine learning research,Atlanta, 6 - 9Nov 2023. PMLR, Vol 229, 711–733.
66.
ShenWYangGYuA, et al. (2023) Distilled feature fields enable few-shot language-guided manipulation. In: TanJToussaintMDarvishK (eds) Proceedings of the 7th conference on robot learning, proceedings of machine learning research, Atlanta, GA, USA, 6–9 November 2023. PMLR, Vol. 229, 405–424.
67.
SinghIBlukisVMousavianA, et al. (2023) Progprompt: generating situated robot task plans using large language models. In: 2023 IEEE international conference on robotics and automation (ICRA), London, UK, 29 May–2 June 2023, 11523–11530.
68.
SridharAShahDGlossopC, et al. (2023) NoMaD: goal masked diffusion policies for navigation and exploration. arXiv pre-print.
69.
StraderJHughesNChenW, et al. (2024) Indoor and outdoor 3D scene graph generation via language-enabled spatial ontologies. IEEE Robotics and Automation Letters9(6): 4886–4893.
70.
SzotACleggAUndersanderE, et al. (2021) Habitat 2.0: training home assistants to rearrange their habitat. In: RanzatoMBeygelzimerADauphinY, et al. (eds) Advances in Neural Information Processing Systems. Curran Associates, Inc., Vol. 34, 251–266.
71.
TouvronHMartinLStoneK, et al. (2023) Llama 2: open foundation and fine-tuned chat models. arXiv.
72.
WahidAStoneAChenK, et al. (2021) Learning object-conditioned exploration using distributed soft actor critic. In: KoberJRamosFTomlinC (eds) Proceedings of the 2020 conference on robot learning, proceedings of machine learning research, Virtual, 16–18 Nov2020. PMLR, Vol. 155, 1684–1695.
73.
WandzelAOhYFishmanM, et al. (2019) Multi-object search using object-oriented pomdps. In: 2019 international conference on robotics and automation (ICRA),Montreal, Quebec, Canada, 20–24 May 2019, 7194–7200.
74.
WangJHeGKantarosY (2025) Probabilistically correct language-based multi-robot planning using conformal prediction. IEEE Robotics and Automation Letters10(1): 160–167.
75.
WaniSPatelSJainU, et al. (2020) Multion: Benchmarking semantic map memory using multi-object navigation. In: LarochelleHRanzatoMHadsellR, et al. (eds) Advances in Neural Information Processing Systems. Curran Associates, Inc., Vol. 33, 9700–9712.
76.
WerbyAHuangCBüchnerM, et al. (2024) Hierarchical open-vocabulary 3D scene graphs for language-grounded robot navigation. Robotics: Science and SystemsXX: 077.
77.
WuSCWaldJTatenoK, et al. (2021) Scenegraphfusion: incremental 3D scene graph prediction from RGB-D sequences. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), New Orleans, LA, USA, 18–24 June 2022, 7515–7525.
78.
XiaFR ZamirAHeZY, et al. (2018) Gibson env: real-world perception for embodied agents. In: Computer vision and pattern recognition (CVPR), 2018 IEEE conference on, Salt Lake City, UT, USA, 18–22 June 2018. IEEE, 9068–9079.
79.
XieQMinSYJiP, et al. (2024) Embodied-rag: general non-parametric embodied memory for retrieval and generation. arXiv.
YadavKRamrakhyaRRamakrishnanSK, et al. (2022b) Habitat-matterport 3D semantics dataset. arXiv preprint arXiv:2210.05633.
82.
YamauchiB (1997) A frontier-based approach for autonomous exploration. In: Proceedings 1997 IEEE international symposium on computational intelligence in robotics and automation CIRA'97. ‘Towards new computational principles for robotics and automation’,Monterey, CA, USA, 10–11 July 1997, 146–151.
83.
YangLKangBHuangZ, et al. (2024) Depth anything: unleashing the power of large-scale unlabeled data. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), Seattle, WA, USA, 16–22 June 2024, 10371–10381.
84.
YeJBatraDDasA, et al. (2021) Auxiliary tasks and exploration enable objectgoal navigation. In: 2021 IEEE/CVF international conference on computer vision (ICCV), Montreal, BC, Canada, 11–17 October 2021, 16097–16106.
85.
YokoyamaNHaSBatraD, et al. (2024) Vlfm: vision-language frontier maps for zero-shot semantic navigation. In: 2024 IEEE international conference on robotics and automation (ICRA),Yokohama, Japan, 13–17 May 2024, 42–48.
86.
YuBKasaeiHCaoM (2023) L3MVN: leveraging large language models for visual target navigation. In: IEEE/RSJ international conference on intelligent robots and systems (IROS),Detroit, MI, USA, Nov6–92023, 3554–3560.
87.
ZhangBSohH (2024) Extract, define, canonicalize: an LLM-based framework for knowledge graph construction. arXiv preprint arXiv:2404.03868.
88.
ZhaoZLeeWSHsuD (2023) Large language models as commonsense knowledge for large-scale task planning. In: OhANaumannTGlobersonA, et al. (eds) Advances in Neural Information Processing Systems. Curran Associates, Inc, Vol. 36, 31967–31987.
89.
ZhengKSungYKonidarisGD, et al. (2020) Multi-resolution POMDP planning for multi-object search in 3D. In: 2021 IEEE/RSJ international conference on intelligent robots and systems (IROS), Prague, Czech Republic, 27 September–1 Oct 2021.
90.
ZhengKPaulATellexS (2023) A system for generalized 3D multi-object search. In: 2023 IEEE international conference on robotics and automation (ICRA), London, UK, 29 May–2 June 2023, 1638–1644.
91.
ZhengHSMishraSChenX, et al. (2024) Take a step back: evoking reasoning via abstraction in large language models. In: KimBYueYChaudhuriS, et al. (eds) International conference on representation learning, Vienna, Austria, 7–11 May 2024, Vol. 2024, 20279–20316.
92.
ZhouGHongYWuQ (2023) Navgpt: explicit reasoning in vision-and-language navigation with large language models. arXiv.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
