Automating urban policy extraction: A large language model-based framework for extracting local heat policies from planning documents

Abstract

Systematic analysis of planning policies is essential for evaluating the coherence and effectiveness of local policy-making, providing critical support for addressing complex urban challenges. However, extracting policies from planning documents is a labor-intensive task, especially when applied at scale across numerous plans. This study develops a novel large language model-based policy extraction (LLM-PE) framework to automate policy extraction and applies it to extracting local heat policies from various planning documents in Miami and Chicago to evaluate its performance. We find that the LLM-PE framework effectively identifies most local heat policies, with an average recall of 79.5%. Additionally, using the newly introduced Human Review Reduction Rate (HRRR) metric, we demonstrate that the average human review workload is reduced by 96.1%. While precision is relatively low (49.5% overall), the results indicate that our framework effectively complements human expertise by pre-screening policies at scale. Technical caveats remain, including the model’s limited ability to detect highly implicit strategies, a challenge exacerbated by inconsistent policy definitions across jurisdictions and plans. We further present a hybrid human–AI workflow for urban policy extraction, enabling scalable policy analytics to support the planning for heat-resilient cities.