Reliable local search is critical for LLM agents, which must locate places that meet user needs at specific times and locations. Existing benchmarks often miss whether results satisfy mandatory constraints, account for preferences, or ground factual and experiential judgments in place attributes and reviews.
We introduce ATLAS-Bench, comprising 1,200 queries from three regions, expanded with real search events, place attributes, reviews, and user profiles. Each query separates mandatory requirements from preferences and factual conditions from subjective judgments, supplying evidence and criteria for verification. Queries are validated through iterative assertion checks and cross‑model review.
We also provide ATLASBox, a standardized evaluation environment equipped with search tools and a fixed point‑of‑interest (POI) knowledge base. Given a query, timestamp, and location, an agent must search attributes and reviews to recommend evidence‑backed options or report that no place meets all mandatory requirements. Evaluation covers answer quality, trajectory quality, and execution cost.
Across 14 configurations of 11 models, GPT‑6‑Luna achieves the highest constraint‑satisfaction rate among recommendations (71.5%) and factual accuracy (95.6%). However, GLM‑5.3 yields a higher acceptable‑recommendation coverage for satisfiable queries (85.3% vs 67.3%), highlighting the difficulty of meeting individual constraints while delivering usable suggestions.
Review