Abstract
Urban challenges necessitate robust evidence synthesis, yet Large Language Models (LLMs) applied to urban studies face factual inconsistencies and a lack of domain-specific grounding. This paper proposes and evaluates a framework for RAG-enhanced LLMs by integrating up-to-date knowledge and policy-relevant evidence into model outputs. Urban Vacant Land (UVL) is selected as the case study, and a systematic literature review (SLR) is conducted to build a retrieval literature base. The results indicate that RAG significantly improves the accuracy of LLMs under structured automatic evaluation. However, although retrieval augmentation provides models with access to domain-specific evidence, its benefits for open-ended planning questions are not consistently reflected in expert ratings, particularly for consistency and creativity. The limited improvement can be attributed to the characteristics of the planning questions, the mismatch between textual information and the spatial data required for urban planning in current RAG pipelines, and potentially ineffective prompting that fails to elicit deeper reasoning. This study contributes to the field by elucidating the capabilities and limitations of LLMs and RAG in urban studies, highlighting that while text-only retrieval is insufficient for context-rich analysis, future advancements in spatially aware hybrid retrieval offer a promising pathway forward.
Keywords
Introduction
Large Language Models (LLMs) have fundamentally advanced natural language processing by leveraging large-scale training data and extensive computational resources to generate coherent and context-aware text (Ge et al., 2023; Yao et al., 2024a, 2024b). Representative models, including the GPT-series and other state-of-the-art LLMs, demonstrate strong capabilities in knowledge encoding, reasoning, and text generation, which have enabled a wide range of applications such as dialogue systems, content creation, and machine translation (Achiam et al., 2023; Austin et al., 2021; Guo et al., 2025; Singhal et al., 2023). Beyond general-purpose language tasks, recent studies have increasingly explored LLM-based solutions in domain-specific contexts, including medicine, human pathology, education, and mathematics, where accurate synthesis of specialized knowledge is often required (Extance, 2023; Lu et al., 2024; Romera-Paredes et al., 2024; Thirunavukarasu et al., 2023). These advances suggest that LLMs could be especially useful for decision-making tasks that require pulling together information from many different sources, reasoning through multiple steps, and providing evidence-based recommendations.
Urban planning represents a canonical example of such settings. Planners synthesize heterogeneous data, from geospatial layers to policy frameworks, to inform spatial interventions and strategic planning. While recent work has begun to apply LLMs to specific urban tasks (Jin and Ma, 2024), a systematic understanding of how to deploy LLMs in a reliable, reproducible, and decision-relevant manner remains limited. In particular, urban planning applications require not only fluent generation but also factual grounding, transparent reasoning, and traceable evidence, which are critical when model outputs may inform real-world planning practices and policy debates.
Recent advances demonstrate the growing influence of LLMs in urban studies. Domain-adapted models and benchmarking frameworks such as CityGPT, Urban Video-Bench, and Open3DVQA have highlighted the potential of LLMs and multimodal language models for urban semantics understanding, spatial reasoning, and navigation-related tasks in complex urban environments (Feng et al., 2025; Zhang et al., 2025a; Zhao et al., 2025). Moreover, LLM-based urban frameworks such as MetaCity have been proposed to support sustainable development goals (SDGs) by leveraging large-scale urban data for simulation, resource allocation, and the identification of potential interventions for congestion, air pollution, and inequality (Zhang et al., 2025b). In parallel, agent-based simulation systems powered by LLMs, such as AgentSociety, illustrate how generative models can be used to simulate large-scale social interactions and explore policy-relevant mechanisms at scale (Piao et al., 2025). Collectively, these studies indicate that LLMs can expand the methodological toolkit of urban research. However, they also highlight an important gap: current applications often focus on showcasing capabilities, while the reliability and evidence-grounding of LLM outputs remain under-examined in planning-oriented knowledge tasks.
Despite their ability to generate human-like text, LLMs may produce responses that are incomplete, inconsistent, or factually incorrect, especially when addressing domain-specific questions that require precise definitions, up-to-date evidence, or nuanced expert judgment (Azaria and Mitchell, 2023; Bang et al., 2023; Zhou et al., 2024). This limitation is particularly problematic for urban planning, where decisions depend on credible evidence and where incorrect statements may mislead analysis or recommendations. Retrieval-Augmented Generation (RAG) offers a promising strategy to mitigate these limitations by coupling LLM generation with external evidence retrieval (Borgeaud et al., 2022; Gao et al., 2023; Jiang et al., 2023; Lewis et al., 2020). In an RAG framework, relevant passages or documents are retrieved from a curated knowledge base and are then provided to the LLM as contextual grounding, thereby improving factual accuracy, domain specificity, and interpretability of responses (Borgeaud et al., 2022; Chen et al., 2024; Wu et al., 2024). However, the performance of RAG critically depends on the quality, coverage, and reliability of its underlying knowledge base, which in turn raises an important methodological challenge: how to construct an evidence-grounded and domain-relevant retrieval corpus for urban planning tasks.
Systematic Literature Review (SLR) provides a structured and reproducible approach to collecting, screening, and synthesizing prior research, thereby enabling the construction of a high-quality domain knowledge base suitable for retrieval-based systems (Cocchia, 2014; Nightingale, 2009). Compared with ad hoc web-scale retrieval, an SLR-driven corpus can offer stronger guarantees of credibility, traceability, and methodological transparency, making it particularly suitable for evidence-intensive domains such as urban planning. Integrating SLR with RAG offers a promising pathway for developing more reliable LLM-based decision-support tools for urban researchers and practitioners (Fu, 2024; Hou et al., 2025; Zhang et al., 2025c).
Against this background, this study evaluates the performance of several widely used LLMs and examines whether they can approximate expert-level performance on research questions in urban studies. We select Urban Vacant Land (UVL) as the focal topic to test this capability because it is a planning-relevant issue that requires synthesizing heterogeneous evidence (e.g., land-use concepts, spatial patterns, governance strategies, and empirical findings) and remains under-explored in both research and practice. Leveraging an up-to-date and curated research database, we implement an RAG-based workflow to ground model outputs in credible literature and compare responses across models and settings (Chen et al., 2024; Gao et al., 2023; He et al., 2025; Lyu et al., 2025). We pose a set of challenging and under-explored UVL questions and ask domain experts to evaluate model outputs in terms of depth, relevance, factual grounding, and their potential to support urban research and planning decisions. By doing so, this study contributes to a systematic understanding of when and how LLMs, especially when augmented by retrieval, can be reliably used to generate evidence-grounded knowledge in UVL and, more broadly, in data-driven urban planning contexts.
Materials and methods
To ensure the effectiveness of the RAG framework, building a reliable knowledge base is essential, as the quality of retrieved information directly influences the accuracy and coherence of generated responses. The method framework consists of three key steps: (1) knowledge database via SLR, (2) construction of UVL vector database, and (3) RAG-based knowledge generation. The framework of this research is shown in Figure 1.
The framework of this research. The first step is utilizing the SLR method to collect specific documents for constructing a retrieval knowledge base. In the second step, these documents are transformed into vectors to build a UVL vector database. The third step illustrates how user queries are processed through the RAG framework to generate responses based on the UVL vector database.
Knowledge database via SLR
To support evidence-grounded urban planning knowledge generation, this study first constructed a domain-specific retrieval corpus using a Systematic Literature Review (SLR) protocol. The standard SLR framework followed a structured process, including several steps: formulate the problem, search the literature, screen for inclusion, extract data, analyze and synthesize data, and report findings (Ludvigsen et al., 2016; Xiao and Watson, 2017). The full-text documents retrieved through these processes comprehensively addressed the research questions and demonstrated substantial scholarly value (Bramer et al., 2017).
This research employs the SLR framework to systematically collect peer-reviewed academic works and case studies on UVL. Specific steps are shown in Figure 2 and described as follows. Furthermore, a format conversion step is added to support the next RAG part. The objective is to establish a specialized retrieval database that enhances the capacity of LLMs to process domain-specific UVL queries with precision and contextual relevance.
Workflow of knowledge database via systematic literature review (SLR). This figure illustrates the process of identifying and preparing domain-specific literature for constructing the UVL retrieval database. Starting from defining research questions related to UVL, relevant keywords are used to search the Web of Science Core Collection. All searched documents are evaluated by LLMs and a domain expert. Full texts are then downloaded and processed using PDF-Extract-Kit models and converted into plain text.
Define research scope and question
Research questions serve as the foundational driver of the systematic literature review process (Kitchenham and Charters, 2007). We scope the review around “Urban Vacant Land (UVL)” as a planning-relevant topic that requires synthesizing evidence across conceptual definitions, identification approaches, spatial mechanisms, and governance interventions. Urban Vacant Land (UVL) has emerged as a global issue closely tied to sustainable development and population shrinking at both regional and international levels (Tu et al., 2024; Wang and Long, 2023, 2026). Prior research has examined UVL from multiple perspectives, including land-use policy, public administration, and community regeneration (Gao and Ryan, 2021; Nassauer and Raskin, 2014; Zhu et al., 2023), and has proposed a variety of planning and management strategies for policymakers and other stakeholders (Bowman and Kim, 2016; Kim et al., 2015; López et al., 2021; Park et al., 2021). When addressing UVL and related city challenges, planners and officials often draw on a broad range of literature, case studies, and professional judgment to identify potential solutions and chart future development pathways, due to the heterogeneous and context-specific nature of evidence in the field. (Khavarian-Garmsir, 2023; Pallagst et al., 2009). Meanwhile, recent advances in natural language processing have enabled the emerging use of LLMs for literature-driven synthesis and planning-related knowledge tasks, highlighting the importance of building curated and traceable evidence bases for domain-specific applications (Fu, 2024).
Design search strategy
A comprehensive search strategy was developed to systematically identify relevant studies. Searches were conducted in the Web of Science Core Collection in December 2024 using topic-based queries (title, abstract, and keywords). We used a combination of UVL-related terms, including “urban vacant land”, “urban vacant lot”, “unused urban spaces”, and “vacant land in city”. Web of Science was selected due to its broad coverage of peer-reviewed literature across urban planning and related disciplines, supporting a cross-disciplinary evidence base for UVL synthesis (Bramer et al., 2017; Lim et al., 2019; Ruhlandt, 2018; Xiao and Watson, 2017).
Search in database
The search returned a total of 1,205 records, including 1,088 journal articles, 32 books/book chapters, and other document types indexed in Web of Science. For each record, bibliographic metadata (title, authors, abstract, DOI, and publication details) were exported for subsequent screening and analysis. Records were exported in plain-text and BibTeX formats, and duplicates were removed prior to screening based on DOI and title matching. The full retrieval list is provided in the Supplementary Table 1.
Screen for inclusion
This step aimed to determine whether each retrieved record should be included in the UVL knowledge database, which serves as the retrieval corpus for the subsequent RAG workflow (Brereton et al., 2007; Xiao and Watson, 2017). Given that the selected papers span a wide range of research fields related to UVL, this step ensures that each paper specifically addresses UVL-related topics for tackling UVL challenges.
To improve screening efficiency, we employed multiple state-of-the-art Large Language Models (LLMs) as assistive reviewers to perform abstract-level relevance classification (Achiam et al., 2023; Grattafiori et al., 2024; Guo et al., 2025; Hurst et al., 2024; Team et al., 2024; Yang et al., 2024). Four models, including GPT-3.5-turbo, Llama-3.1-405B, GPT-4o, and Claude-3.5-Sonnet, all of which are widely recognized for their advanced natural language understanding and analytical capabilities, were queried via API using a unified prompt template (Anthropic, 2024; Grattafiori et al., 2024; Hurst et al., 2024). The prompt defined UVL as abandoned, underutilized, or undeveloped parcels of land within urban areas and asked each model to return a binary judgment (1 = eligible / UVL-relevant, 0 = not eligible) based on whether the abstract contains evidence aligned with the inclusion criteria above.
The determined options utilized in this research for requesting results from these LLMs are as follows.
–You are an urban researcher specializing in the identification and analysis of urban vacant land (UVL).
–Urban vacant land refers to abandoned, underutilized, or undeveloped parcels of land within urban areas.
–Your task is to determine whether the provided abstract is relevant to UVL research, including but not limited to topics such as UVL definition, identification, causes, development mechanisms, advantages and disadvantages, solutions, and related issues.
–Please answer with 1 (yes) or 0 (no).
–The abstract is as follows: {Abstract}.
The “{Abstract}” indicates the abstract text of each paper in the first round selection. To facilitate this determining process, the models were designed to provide binary responses. “1” (yes) if the paper addressed a UVL solution, and “0” (no) otherwise. Each of the four LLMs was queried via API using these prompts. The use of these multiple LLMs enabled cross-validation to compare the outputs of different models, ensuring robustness and consistency in the results.
In parallel, an urban planning expert independently reviewed the same set of 1,205 abstracts. To ensure transparent and reproducible screening, we computed pairwise agreement rates among the four LLMs and the human expert based on Cohen’s kappa. The interpretation of Cohen’s kappa followed the criteria of Landis and Koch (1977), where values representing agreement are categorized as slight (0.00–0.20), fair (0.21–0.40), moderate (0.41–0.60), substantial (0.61–0.80), and almost perfect (0.81–1.00). As shown in Figure 3, the kappa values ranged from 0.57 to 0.82, and the specific results are as follows: GPT-3.5 (0.74), LLaMA-3.1 (0.62), GPT-4o (0.72), and Claude-3.5-Sonnet (0.82). The highest agreement was observed between Claude-3.5-Sonnet and human experts (0.82), while the lowest was between LLaMA-3.1 and Claude-3.5-Sonnet (0.57). Final inclusion decisions were made by synthesizing the LLM recommendations and expert judgment using a majority voting scheme (Chiang et al., 2024). To reduce the risk of false exclusions, records with low agreement across reviewers were subject to expert adjudication prior to full-text retrieval. Specifically, a paper was considered to meet the selection criteria for the next step if at least three out of the five results returned the same decision.
Pairwise agreement rates among four LLMs and the human expert. This figure shows the Cohen’s kappa among four LLMs (GPT-3.5, LLaMA-3.1, GPT-4o, Claude-3.5-Sonnet) and human expert during screening 1,205 records to 634 papers.
Format conversion
After screening, full texts of the included studies were downloaded in PDF format and converted into machine-readable plain text to support document retrieval and evidence attribution in the subsequent RAG pipeline (Radford et al., 2019). We used the GPU-accelerated MinerU tool (Wang et al., 2024) to extract structured elements (e.g., titles, paragraphs, and tables) while removing non-textual content (e.g., figures) to reduce retrieval noise. Each document was assigned a unique identifier and linked to its DOI to ensure traceability between retrieved text chunks and their original sources. In total, 634 full-text papers (from the 1,205 retrieved records) were processed and prepared for semantic embedding and retrieval (see Supplementary Table 1).
Construction of UVL vector database
To enable retrieval-augmented generation, we transformed the curated UVL full-text corpus into a vector database for semantic search. Each document was segmented into text chunks to support fine-grained retrieval while preserving contextual coherence (Gong et al., 2020). Chunking was performed in a structure-aware manner to retain logical boundaries (e.g., paragraph and section structure), and each chunk was stored together with metadata, including document ID, DOI, publication year, and section label, to support traceable evidence attribution during generation.
All chunks were embedded into dense vector representations using the Snowflake Arctic-Embed 2 model (Yu et al., 2024). Embeddings capture the semantic meaning of each chunk and enable similarity-based retrieval for planning-relevant queries. Model configuration details (e.g., chunk length and overlap) are reported in the Supplementary Materials to ensure reproducibility.
The resulting embeddings were indexed and stored in LanceDB, which supports approximate nearest neighbor search for efficient retrieval of semantically related evidence fragments. During inference, user queries are embedded using the same model and matched against the UVL vector database to retrieve the top-k most relevant chunks, which are then provided to the generator as contextual grounding in the downstream RAG pipeline.
RAG-based knowledge generation
Based on the UVL vector database and embedding model described above, we implemented a Retrieval-Augmented Generation (RAG) workflow to support evidence-grounded UVL knowledge generation. RAG enhances a language model by conditioning its responses on passages retrieved from an external corpus, thereby improving domain specificity and reducing unsupported claims when answering planning-relevant questions (Wu et al., 2024). As illustrated in Figure 1, the workflow consists of three main steps.
First, a user query is encoded into a dense vector using the same embedding model (Snowflake Arctic-Embed 2) applied to the corpus. This representation captures semantic meaning and enables similarity-based retrieval beyond exact keyword matching. Second, the query vector is used to retrieve the top-ranked text chunks from the UVL vector database. The retrieved evidence is then assembled into a “query + context” prompt, preserving links to document identifiers and DOIs to facilitate evidence attribution. Third, the composed prompt is provided to a generative LLM to produce a final response. Unless otherwise specified, ChatGPT-4o-mini was used as the default generator in our implementation. By grounding generation in retrieved literature, this RAG workflow can improve factual consistency and transparency relative to direct generation without retrieval, while allowing outputs to be traced back to supporting sources within the curated UVL evidence base.
Evaluation of RAG-supported LLMs
Evaluating LLMs is inherently task-dependent, and the choice of evaluation criteria should reflect domain-specific requirements (Chang et al., 2024; Hendrycks et al., 2021). In urban planning and urban studies, useful model outputs are expected to be not only fluent and relevant but also conceptually sound and supported by credible evidence. Following recent methodological guidance on LLM evaluation, we adopted a two-part evaluation framework combining automatic evaluation and expert-based human evaluation (Chang et al., 2024; Zhong et al., 2022), as illustrated in Figure 4.
Evaluation framework for assessing RAG-based knowledge generation. This figure presents a two-part evaluation framework to assess the performance of RAG models. In the automatic evaluation, LLM-generated answers are compared against expert-annotated references using standard metrics such as F-score, Recall, and Precision. In the human evaluation, reviewers assess RAG outputs across multiple topics using several indicators to provide comprehensive assessments.
We conducted a comparative evaluation across five widely used LLM families (Qwen2.5, Gemma3, Llama3.1, Deepseek-R1, and ChatGPT-4o). For each family, we tested two representative model sizes to examine scaling effects. To evaluate the contribution of retrieval grounding, we further integrated RAG with the larger model variants, resulting in 15 model configurations in total (Table 1). Unless otherwise specified, all models were evaluated under consistent prompting settings, and RAG used the same UVL Systematic Evidence Base described in Section 2.1.
List and cost of different models. The data presented in this table is sourced from cloud infrastructure websites as of April 2025.
Automatic evaluation
Automatic evaluation provides a scalable approach for benchmarking model performance on structured tasks (Chang et al., 2024). Because standardized benchmark datasets for urban planning applications remain limited, we designed a UVL evidence recognition task to evaluate whether LLMs can correctly identify UVL-relevant academic content. Specifically, we sampled 50 paragraphs from UVL-related and non-UVL research articles and asked models to classify each paragraph as UVL-relevant (“Y”) or not (“N”), accompanied by brief reasoning. Expert annotations served as reference labels. Model outputs were assessed using standard information-retrieval metrics, including Precision, Recall, and F-score.
Human evaluation
Human evaluation is essential for assessing open-ended, planning-relevant knowledge tasks where model usefulness depends on evidence-grounded reasoning, completeness, and decision relevance (Chang et al., 2024; Lee et al., 2023). To verify whether large language models can generate new knowledge in urban studies, we focused on multi-stakeholder topics covering government and resident perspectives, uncertainties from demographic shifts and technological advancements affecting UVL, and environmental change factors. Such themes have attracted considerable scholarly interest in recent studies (Tu et al., 2024; Wang and Long, 2026). Accordingly, we constructed the following five UVL-oriented research questions (Table 2). Topic 1 (Planning Strategies) examines practical governance responses from a governmental perspective, exploring how to rapidly convert vacant land into emergency facilities during crises while overcoming property-rights coordination and legal barriers (Agheyisi, 2025). Topic 2 (Community Psychology) addresses the community perspective by evaluating residents’ perceptual thresholds and cross-cultural differences in tolerance that shape acceptance of UVL interventions (Rupp et al., 2022). Topic 3 (Technological Development) focuses on the future technological perspective by assessing the integration of big data and AI for forecasting vacant land demand and potential uses under rapid urbanization (Wang and Long, 2026). Topic 4 (Ecological Effects) considers the ecological perspective by quantifying and maximizing the ecological value of UVL, including biodiversity conservation, rainwater management, and carbon sequestration (Chen et al., 2026). Finally, Topic 5 (Population Changes) investigates the demographic dimension by analyzing how urban population loss influences the spatial patterns and scale of vacant land and identifying critical intervention thresholds (Tu et al., 2024; Wang and Long, 2023). By structuring these five questions, we evaluate the models' ability to synthesize heterogeneous and context-dependent evidence, thereby enhancing the scope, generalizability, and practical relevance of our human evaluation results.
Topics and detailed prompts for testing performances of generating new knowledge.
Reviewers assessed responses using eight criteria adapted from prior LLM evaluation frameworks (Liang et al., 2022; OpenAI, 2024; Van Der Lee et al., 2019): accuracy, relevance, depth/completeness, consistency, readability, usefulness, creativity, and safety (Table 3). To reduce subjective bias, reviewers rated each criterion independently in separate passes, and each topic was evaluated by two reviewers blinded to model identity. Final scores were computed as the average of the two reviewers’ ratings. All model responses and evaluation materials are provided in the Supplementary Materials.
The reviewer metric for evaluating the performance of LLMs.
To ensure inter-rater reliability, a pilot grading session was conducted where all reviewers cross-evaluated a subset of responses to align scoring standards. The inter-rater agreement was monitored, and for any score discrepancy greater than 2 points, a third senior expert intervened to facilitate a consensus, ensuring the robustness of the human-centric assessment.
Results
After SLR screening and full-text processing, 634 papers were included in the UVL Systematic Evidence Base and further prepared for retrieval-augmented experiments. The RAG workflow was implemented using AnythingLLM, which provides an integrated pipeline for document ingestion, embedding, vector indexing, and retrieval-based context assembly. Text embeddings were generated using Snowflake Arctic-Embed 2, deployed via Ollama. The software environment was configured with CUDA 11.8 and PyTorch 2.3.1.
During inference, user queries were first embedded and used to retrieve semantically relevant chunks from the UVL vector database. Retrieved evidence passages were then assembled into a “query + context” prompt and passed to the target LLM for response generation. Smaller open-source models (Qwen2.5-7B, Llama3.1-8B, Deepseek-R1-7B, and Gemma3-4B) were executed locally on a GPU, whereas larger models were accessed via cloud inference services due to compute constraints. To ensure comparability, all models were evaluated using consistent prompts and generation settings. A representative example of an evidence-grounded Q&A interaction is shown in Figure 5.
An example of RAG+LLM-generated response to user query.
Representativeness test
Preliminary descriptive statistics suggest that UVL-related studies are predominantly concentrated in North America, East Asia, and Europe, with comparatively fewer contributions from Africa and South America. The publication timeline is skewed toward the period after 2010, consistent with rising scholarly interest in UVL under the broader agenda of sustainable urban development. Most included sources are drawn from high-impact journals in urban planning, environmental science, and geography indexed in the Web of Science Core Collection. Although this distribution reflects the current structure of peer-reviewed UVL research, it may introduce regional and temporal biases into the retrieval corpus, potentially shaping model outputs toward context-specific evidence. Future work will broaden coverage by incorporating additional bibliographic databases, grey literature, and multilingual sources to improve representativeness and reduce systematic omission.
Automatic evaluation results
Large-size models, as expected, achieved higher accuracy but at much lower generation speeds, consistent with prior observations in large-model deployment settings (Feng et al., 2025; Ma et al., 2025). However, in this UVL evidence recognition task (Figure 6), scaling model size brought only a modest improvement in baseline F-score, raising it from 0.47 to 0.53 (+13.33%). It is important to interpret the speed metrics with caution because the small-sized models were benchmarked on a local GPU, whereas the large-sized models were evaluated via cloud inference services, where throughput can be affected by network latency and platform variability. Moreover, GPT-4o-mini, as a closed-source model that cannot be deployed locally, was assessed exclusively via its API.
Automatic evaluation result of different LLMs. The top three charts display the F-score, precision, and recall of each model, while the bottom three show generation speed (tokens per second). Black bars indicate the mean values with 95% confidence intervals. The p-value between the Large Size Model and the RAG+ Large Size Model is 0.0025, which means the improvement is significant.
The largest improvement was observed when integrating RAG with the large-sized models (Figure 6), which increased the F-score from 0.53 to 0.73 (+38.67%). In the non-RAG setting, GPT-4o-mini performed best among the small-sized models, and Gemma3-27B achieved the highest score in the large-sized group, with GPT-4o and Deepseek-R1-70B showing comparable performance. Under RAG augmentation, Qwen2.5-72B and Llama3.1-70B exhibited substantial gains, whereas Gemma3-27B showed limited improvement. Significance testing reveals a p-value < 0.05 for the comparison between Large Size Model and RAG+Large Size Model, indicating that the improvements provided by RAG are statistically significant. These results suggest that retrieval grounding can materially enhance UVL-related evidence recognition, while the magnitude of gains varies across model families, highlighting practical trade-offs between model scale, deployment constraints, and retrieval effectiveness.
Human evaluation results
Human evaluation revealed a different performance pattern from the automatic benchmark, and the detailed results are reported in Figure 7 and Table 4. While the automatic evaluation suggests that model scale and retrieval grounding can influence UVL evidence recognition performance, reviewers did not consistently rate RAG-augmented configurations higher than their non-RAG counterparts. Specifically, the Average for All Large Size Model group (7.01) outperformed the RAG+Large Size Model group (6.76). For several models (e.g., Qwen2.5-72B, Llama3.1-70B, Gemma3-27B, and GPT-4o), mean human-rated scores were slightly lower with RAG, whereas Deepseek-R1-70B showed a modest improvement.
Radar charts of human evaluation results. Each radar chart demonstrates eight indicators for each model. The final results for all indicators are presented following the model’s name.
Human evaluation results of different LLMs.
Note. All indicators were rated on a 1–10 scale. Abbreviations: Acc. = Accuracy; Rel. = Relevance; D/C = Depth/Completeness; Cons. = Consistency; Read. = Readability; Use. = Usefulness; Cre. = Creativity; Saf. = Safety; Ave. = Average score across all indicators for each model.
Across all model configurations, the Group Average ranged from 6.71 to 7.01, indicating that experts generally perceived the responses as fair to fair-to-good. For each indicator, Creativity (mean = 5.96) and Depth/Completeness (mean = 6.42) received the lowest ratings, suggesting persistent limitations in generating conceptually rich and non-trivial answers to planning-oriented research questions. In contrast, Readability, Accuracy, and Consistency were rated relatively higher, implying that current LLMs can produce fluent and logically structured responses even for specialized UVL prompts.
Examples illustrate this pattern. Comparing the Llama3.1-70b and its RAG-enhanced version (RAG+Llama3.1-70b) on Topic 1 (see Supplementary Material Page 12 and Page 23 for details), the Llama3.1-70b outlines concepts like Dynamic Governance Frameworks and Collaborative Partnerships, while the RAG model adds finer-grained tools such as Temporary Use Permits and Community Land Trusts. However, both responses are fragmented and piecemeal, making it difficult to discern the relationships among the various strategies. They lack a coherent, complete logical chain for addressing the temporary use of vacant urban land. For instance, it remains unclear what types of communities or policies could implement community land trusts or what specific problems they are intended to solve. A comparison between the DeepseekR1-70b and its RAG-enhanced counterpart on Topic 2 (see Supplementary Material, Pages 43 and 52) reveals a similar issue. The base model mentions location, cultural influences, economic factors, and other elements, while the RAG version shows minimal structural or substantive divergence from it, further underscoring that RAG does not yield a clear improvement in answer quality.
One possible explanation is that retrieved passages may introduce redundant or loosely related context, which can dilute answer focus or reduce perceived originality, especially when prompts do not explicitly require citation-based reasoning. Another consideration is the design of the prompts themselves. The instructions provided to the models may not have been sufficiently detailed or directive to guide the RAG systems toward synthesizing retrieved information into novel, insightful, or decision-relevant arguments, rather than simply listing or summarizing facts. Overall, these findings suggest that, under our evaluation setting, retrieval grounding does not automatically translate into improved human-perceived usefulness or insightfulness. Although LLMs can effectively organize existing knowledge into coherent narratives, they remain less reliable at producing deeply reasoned, creative, and decision-relevant syntheses that go beyond surface-level summarization of retrieved evidence.
Discussion
Large Language Models (LLMs) and retrieval-augmented generation (RAG) are increasingly used in urban research to support knowledge synthesis and decision-oriented analysis. Building on this trend, we constructed a UVL Systematic Evidence Base through a transparent Systematic Literature Review (SLR) pipeline and integrated it into an RAG workflow to evaluate whether widely used LLMs can generate evidence-grounded responses to planning-relevant UVL questions. By benchmarking multiple model families under both non-RAG and RAG settings, this study provides a structured assessment of LLMs as potential research assistants in urban studies.
Overall, our results demonstrate that RAG can improve performance under structured automatic evaluation, highlighting the value of a domain-specific evidence base for UVL-related tasks. These gains reinforce the methodological premise that RAG performance is highly sensitive to the quality, coverage, and representativeness of the underlying retrieval corpus. Compared with web-scale retrieval, an SLR-based corpus offers stronger transparency and traceability, as retrieved passages can be linked back to specific papers and DOIs, enabling expert verification and more reliable evidence attribution.
However, human evaluation revealed a more nuanced pattern. Although retrieval augmentation provides models with access to domain-specific evidence, its benefits for open-ended planning questions were not consistently reflected in expert ratings (Figure 7). As shown in Figure 8, RAG exhibits a slight negative impact on model performance across both dimensions and topics, though these effects are not statistically significant (p-values range from 0.180 to 0.874 by dimension and from 0.337 to 0.810 by topic). This indicates that for open-ended, domain-specific questions, expert evaluations do not demonstrate a clear improvement of RAG-enhanced models over their base counterparts.
Human evaluation results by Dimension and Topic. The RAG Effect Size by Dimension panel (upper) shows the mean score difference (RAG minus non-RAG) for each quality dimension, with 95% confidence intervals. The RAG Performance by Topic panel (lower-left) displays the RAG Advantage Score shift across the five topics. The Model Performance Ranking panel (lower-right) compares the average scores of all models. No statistically significant improvement from RAG enhancement is observed (dimension-level p-values: 0.180-0.874; topic-level p-values: 0.337-0.810).
Some domain-specific factors help explain this pattern. First, urban planning questions do not have a single standardized “correct answer.” Instead, appropriate responses are contingent on stakeholder priorities and local conditions, and different analytical perspectives can legitimately lead to different conclusions. Urban studies is therefore a challenging domain for LLM-based assistance because its evidence base is inherently heterogeneous, multi-scalar, and context-dependent. Answering planning-oriented inquiries typically requires integrating fragmented evidence across conceptual debates (e.g., how “urban vacant land” should be defined), methodological diversity (remote sensing detection, parcel-level records, case studies, and governance analyses), and normative as well as practical constraints such as equity, feasibility, and legal legitimacy.
Second, most current RAG pipelines are built around text semantics, whereas urban research concerns a coupled spatial-social system; this modality mismatch can impose bottlenecks on retrieval and reasoning for planning tasks. Text-similarity retrieval alone often cannot represent spatial semantics (e.g., topology, distance, and spatial context), nor can it reliably integrate heterogeneous geospatial information (vector/raster data and coordinate reference systems) that underpins many UVL analyses (Crooks and Chen, 2024; Li et al., 2024; Liang et al., 2025, 2026). In addition, planning-relevant synthesis depends on domain-specific concepts and subtle human behavior patterns, which may be poorly captured by narrow topic-specific retrieval (Crooks and Chen, 2024; Xu et al., 2026).
Third, the prompts used in our evaluation may not have been fully optimized to elicit the LLMs’ latent reasoning capabilities. The formulation of the task instructions can critically influence how retrieved knowledge is utilized; a prompt that fails to explicitly call for creative integration, multi-perspective analysis, or forward-looking synthesis may result in outputs that merely echo the retrieved content rather than leveraging it to generate novel insights. Future work should explore prompt engineering strategies that better scaffold the transition from evidence retrieval to generative reasoning, thereby unlocking the full potential of RAG-enhanced models in complex planning contexts.
From a practical standpoint, these findings carry implications for evidence based planning. Compared with factual accuracy or retrieval speed, urban planners and decision-makers require depth of analysis, conditional reasoning, and the ability to synthesize weakly related contextual information across spatial, social, and institutional dimensions. The current RAG framework, by prioritizing narrowly relevant passages from the UVL corpus, may under-emphasize such auxiliary information, thereby limiting its immediate applicability in complex, high-stakes planning scenarios. This highlights that retrieval augmentation alone is insufficient; future systems must explicitly balance retrieval precision with synthesis breadth to deliver genuinely decision-relevant insights (Jiang et al., 2025; Xu et al., 2025).
Beyond improving retrieval grounding for a single task, the proposed SLR-RAG framework can be viewed as an initial step toward a broader, auditable evidence infrastructure for urban research. In many planning subfields, core knowledge remains dispersed across disciplines and publication venues, making it difficult to trace how concepts, mechanisms, and interventions accumulate over time. By systematically screening full-text literature into a machine-readable and traceable corpus, our framework enables a new pipeline in which evidence can be queried, audited, and compared at scale. The greatest opportunity lies not merely in embedding more papers, but in enriching the evidence base with structured representations, such as consistent tags for city/regional context, governance settings, intervention types, outcomes, and causal mechanisms (Xu et al., 2026).
Several limitations and future directions follow from this work. First, our evidence base was derived exclusively from Web of Science-indexed peer-reviewed English-language literature, which inherently reflects regional and temporal publication biases. High-income countries in North America, Europe, and East Asia are likely over-represented, while studies from the Global South, non-English publications, and older foundational work are under-represented. This limits the current generalizability of our conclusions to global urban planning practice. Second, retrieval design and evaluation need tighter alignment: future work should incorporate explicit evidence-grounding criteria (e.g., citation faithfulness) and retrieval-aware prompting to ensure that retrieved passages are used as support rather than distraction. Third, while LLM-based agents and simulation frameworks hold promise for policy exploration, their deployment in high-stakes planning contexts requires careful attention to transparency, accountability, and scenario-based validation. Finally, longitudinal and cross-context evaluations across diverse urban settings are needed to clarify when RAG-enhanced LLMs can serve as reliable research assistants and when expert judgment remains indispensable. To address the observed modality mismatch and knowledge-constraint effect, future iterations could incorporate hybrid retrieval strategies, such as multi-hop retrieval for indirect spatial-social relationships, geospatial-aware embeddings, or adaptive relevance thresholds that deliberately include weakly related contextual sources (Liang et al., 2025, 2026; Xu et al., 2026).
Conclusions
This study proposes a systematic framework that integrates RAG with LLMs to support planning-oriented urban research, using a systematically curated Urban Vacant Land (UVL) evidence base. By benchmarking multiple widely used LLMs under both non-RAG and RAG settings, we show that retrieval grounding can substantially improve performance on fact-oriented tasks in our automatic evaluation, particularly for yes-or-no evidence recognition. However, human evaluation indicates that these gains do not consistently translate into higher perceived usefulness or insightfulness. Despite producing fluent and coherent responses, current models still struggle to generate mechanistic reasoning and genuinely creative, decision-relevant insights comparable to expert judgments.
Overall, our findings highlight both the opportunities and limitations of RAG-enhanced LLMs as urban research assistants. A curated and traceable literature-based corpus can strengthen evidence grounding and transparency, yet stronger prompting, retrieval calibration, and evaluation protocols are needed to ensure that retrieved evidence supports, not distracts from, planning-relevant synthesis. Future work should expand the evidence base coverage and develop retrieval-aware evaluation criteria to better assess when and how LLM systems can reliably support urban policy analysis and regeneration strategies.
Footnotes
Acknowledgements
The authors would like to thank the five independent urban researchers for their time and expertise in evaluating the model outputs. We also appreciate the support and advice provided by Dr. Ying Long. The main research work and manuscript preparation are completed at Tsinghua University, while the submission and subsequent revisions are carried out at South China University of Technology.
Ethical considerations
All expert reviewers participated voluntarily and were fully informed about the research objectives and the evaluation process. This study involved an expert evaluation of AI-generated text and did not collect sensitive personal data.
Funding
The author received no financial support for the research, authorship, and/or publication of this article.
Declaration of conflicting interests
The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Data availability statement
The systematic retrieval list and all LLM-generated responses along with expert evaluation scores are available in the Supplementary Materials.