An email assistant should not do less work just because a user phrases the same request differently. Most benchmarks, however, test each task with only one canonical request, leaving robustness to linguistic variation largely unmeasured. We investigate whether assistants stay reliable when the requested information, available evidence, and expected outcome stay fixed but the communication style or English variety changes. We create validated variants along five style axes and four rule‑based dialect conditions, then evaluate them on three benchmarks: a retrieval‑augmented generation (RAG) pipeline and two tool‑using agents. Indirect requests reduce performance on all three benchmarks, while formal requests hurt performance on both agentic benchmarks. A closer look shows that verbose requests mainly impair a lexical retriever by making the relevant email harder to find. By contrast, indirect and dialect variants remain harmful even when the relevant email is retrieved. In the agentic setting, indirect and formal requests cause the agents to omit required actions rather than to take extra unsupported actions. These findings indicate that a seemingly successful response is not enough to claim robustness: evaluations should vary how requests are expressed and separately measure whether agents actually complete the requested work.
Review