Large language models (LLMs) have shown remarkable performance on standard benchmarks, yet it remains largely unexplored whether they truly meet user expectations.
Existing evaluation methods relying on model heuristics, expert rubrics, or user simulations fail to capture the diversity and nuance of real human expectations, causing models to seem competent while misaligning with what users actually seek.
This study presents the first systematic investigation of user expectations in real-world LLM interactions, proposing a principled procedure to extract semantically rich expectations and introducing ExpectBench, a benchmark grounded in real user expectations.
Analyses reveal that current LLMs struggle to satisfy and anticipate what users hope to obtain, highlighting a fundamental source of misalignment.
Building on these observations, we propose LENS, a lightweight latent expectation-aware response generation framework. LENS enables models to internalize user expectations and generate better-aligned responses, consistently improving expectation satisfaction and underscoring the importance of explicitly modeling user expectations for realistic human-AI alignment.
Blogger's Review: This paper delves into the misalignment between LLMs and user expectations, proposing ExpectBench and the LENS framework as innovative solutions. It emphasizes the critical importance of understanding real user needs, which could significantly advance human-computer interaction research.