In urban simulation, LLM-based generative agents are increasingly utilized, yet it remains unclear whether they replicate empirically realistic human mobility patterns. To address this, we introduce a validation framework for evaluating the mobility of generative agents in LLM-based urban simulators against real-world mobility data. We utilize mobility laws, temporal rhythms, network motifs, semantic activity transitions, and behavioral mobility profiles.
By evaluating datasets from the Greater Paris region and Shanghai, we analyze AgentSociety and CitySim across multiple dimensions of mobility realism. Our analysis reveals a substantial gap between narrative plausibility and empirical mobility realism. While the simulators capture some high-level semantic activity distributions, they struggle to reproduce core spatial and temporal constraints, including realistic trip-length distributions, origin-destination flows, dwell times, and transition dynamics.
We further observe that realistic mobility diversity is unstable across default prompting configurations and may require explicit profile-aware initialization. To support reproducible evaluation, we also contribute scalable and open LLM-driven infrastructure for regional-scale map generation, observability-enhanced simulation, mobility-metric computation, and traffic simulation.
Our findings highlight the need for rigorous empirical validation of LLM-based urban simulators and provide practical tools for building more realistic and reproducible urban simulation systems.
Blogger's Review: This paper delves into the application of LLM in urban simulation, pointing out the current technology's shortcomings in mobility realism and emphasizing the importance of empirical validation, paving the way for future research.