Although some intelligent large-scale models have been proposed recently, due to the diversity and complexity of urban planning tasks, there are still no domain-specific large-scale models tailored to urban planning that support varied tasks. To address this gap, this paper presents the Semantic Multimodal Analysis and Retrieval approach for Planning (SMART-Plan), an innovative multimodal large model that supports complex and diverse urban planning tasks such as domain knowledge questioning, multimodal data retrieval and image-based plan generation. Another key innovation of this model lies in the automated construction of a domain-specific knowledge graph, which combines textual and visual data to represent urban planning entities and their relationships comprehensively. By leveraging the constructed knowledge graph and the designed three-phase domain fine-tuning, the performance of the model was significantly improved across multiple tasks in urban planning, addressing the challenges of fragmented data and specialized terminology in urban planning. Extensive experiments demonstrated that SMART-Plan significantly outperforms existing models in accuracy, logic and professionalism, with an average improvement of 6.25% and 7.2% in knowledge Q&A and image–text Q&A, compared to state-of-the-art methods.
ZhuXLiZWangXJiangXSunPWangX. Multi-modal knowledge graph construction and application: A survey. IEEE Trans Knowl Data Eng2022; 36(2): 715–735.
2.
DevlinJChangM-WLeeKToutanovaK.Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
3.
HuEShenYWallisPAllen-ZhuZLiYWangSChenW.LoRA: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
4.
FanWDingYNingLWangSLiHYinDChuaT-SLiQ. A survey on rag meeting LLMs: towards retrieval-augmented large language models. In: Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining, Barcelona, Spain, 25–29 August 2024, pp. 6491–6501.
5.
GuYTinnRChengHLucasMUsuyamaNLiuXNaumannTGaoJPoonH. Domain-specific language model pretraining for biomedical natural language processing. ACM Trans Comput for Healthcare (HEALTH)2021; 3(1): 1–23.
6.
HartsockIRasoolG. Vision-language models for medical report generation and visual question answering. A review. Front Artif Intell2024; 7: 1430984.
7.
LiCWongCZhangSUsuyamaNLiuHYangJNaumannTPoonHGaoJ. LLaVA-Med: training a large language-and-vision assistant for biomedicine in one day. Adv Neural Inf Proces Syst 202336: 28541–28564.
8.
MetaAI.Llama: open foundation and instruction models from Meta[EB/OL]. https://www.llama.com/, 2024.
9.
HeKZhangXRenSSunJ.Deep residual learning for image recognition. In Proceedings of the 2016 IEEE conference on computer vision and pattern recognition. Las Vegas, NV, USA: IEEE, June 27-July 1, 2016, pp. 770–778.
10.
RadfordAKimJWHallacyCRameshAGohGAgarwalSSastryGAskellAMishkinPClarkJKruegerGSutskeverI. Learning transferable visual models from natural language supervision. In: Proceedings of the 38th international conference on machine learning. Virtual conference: PMLR, July 18–24, 2021, pp. 8748–8763.
11.
LiJSelvarajuRGotmareAJotySXiongCHoiSCH. Align before fuse: vision and language representation learning with momentum distillation. Adv Neural Inf Proces Syst 202134: 694–9705.
12.
VaswaniAShazeerNParmarNUszkoreitJJonesLGomezANKaiserŁPolosukhinI. Attention is all you need. Adv Neural Inf Process Syst2017; 30: 998–6008.
13.
LiuYDingJFuYLiY. Urbankg: an urban knowledge graph system. ACM Trans Intell Syst Technol2023; 14(4): 1–25.
14.
BrownTMannBRyderNSubbiahMKaplanJDDhariwalPNeelakantanAShyamPSastryGAskellAAgarwalSHerbert-VossAKruegerGHenighanTChildRRameshAZieglerDWuJWinterCHesseCChenMSiglerELitwinMGraySChessBClarkJBernerCMcCandlishSRadfordASutskeverIAmodeiD. Language models are few-shot learners. Adv Neural Inf Proces Syst 202033: 1877–1901.
15.
TouvronHLavrilTIzacardGMartinetXLachauxM ALacroixTRozièreBGoyalNHambroEAzharFRodriguezAJoulinAGraveELampleG.Llama: open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
16.
HowardJRuderS.Universal language model fine- tuning for text classification. arXiv preprint arXiv:1801.06146, 2018.
17.
LiJLiDXiongCHoiSH. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Proceedings of the 39th international conference on machine learning. Baltimore, MD, USA: PMLR, 17–23 July 2022, pp. 12888–12900.
18.
AlayracJ-BDonahueJLucPMiechABarrIHassonYLencKMenschAMillicanKReynoldsMRingRRutherfordECabiSHanTGongZSamangooeiSMonteiroMMenickJLBorgeaudSBrockANematzadehASharifzadehSBińkowskiMBarreiraRVinyalsOZissermanASimonyanK. Flamingo: a visual language model for few-shot learning. Adv Neural Inf Process Syst2022; 35: 23716–23736.
19.
LiuHLiCLiYLeeYJ. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. Vancouver, Canada: IEEE, June 16–21, 2024, pp. 26296–26306.
20.
PanSLuoLWangYChenCWangJWuX. Unifying large language models and knowledge graphs: a roadmap. IEEE Trans Knowl Data Eng2024; 36(7): 3580–3599.
21.
National Catalogue Service for Geographic Information [EB/OL]. https://www.webmap.cn/ (accessed 21 July 2025).
LiuYLiuHWangHMengFLiuM. BCAN: bidirectional correct attention network for cross-modal retrieval. IEEE Trans Neural Netw Learn Syst2023; 35: 14247–14258.
28.
LiJLiDSavareseSHoiS.Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv: preprint arXiv:2301.12597, 2023.
29.
BaiJBaiSYangSWangSTanSWangPLinJZhouCZhouJ.Qwen-VL: a frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966. 2023.
