Abstract
As large language model agents are increasingly integrated into map services, users often express their needs informally in everyday scenarios, resulting in underspecified queries with many unspoken needs—namely, implicit decision factors critical for user satisfaction.
While clarification can mitigate this issue, it adds to user burden in daily interactions; thus, a capable agent should proactively recover such factors from available information sources. However, evaluating this ability is challenging.
The first challenge is to determine which implicit decision factors are suitable for evaluation. A factor is evaluable only if it affects user acceptance and can be recovered from information available to the agent before it responds.
Secondly, user satisfaction cannot be reliably represented by a single reference answer, necessitating a benchmark that converts satisfaction-relevant factors into objective and quantifiable evaluation targets.
To address these challenges, we propose a restore-identify-filter framework that reconstructs complete user needs from behavior-chain evidence, identifies implicit decision factors, and retains only those supported by pre-query evidence.
Building on this methodology, we construct MapSatisfyBench from large-scale, real-world anonymized user data and annotate ground truth from five dimensions, enabling full-chain evaluation of satisfaction-aware map agents.
Experiments show that current agents generally perform well on explicit task completion but remain limited in satisfying implicit decision factors and proactively acquiring the evidence needed for satisfaction-aware decisions.
These findings establish MapSatisfyBench as a benchmark for shifting map-agent evaluation from task completion toward satisfaction-aware spatial decision making.
Blogger's Review: This research introduces a novel evaluation framework for user satisfaction in map services, highlighting the significance of implicit decision factors and advancing the state-of-the-art in the field. Future map agents need to proactively extract these factors during user interactions to enhance service quality.