Real‑world business requests often omit key requirements because frontline practitioners find it hard to verbalize the rules they know from experience. Forward‑deployed engineers (FDEs) must clarify these requirements through dialogue and iteratively optimize solutions until they satisfy customer acceptance criteria. Existing optimization benchmarks usually start from explicit objectives, while clarification benchmarks artificially introduce information gaps into existing tasks.
We introduce FDE‑Bench, the first benchmark built from actual customer projects to evaluate agents that need interactive requirement clarification and iterative optimization. The benchmark comprises 49 cases spanning combinatorial optimization and machine learning. Clarification targets are distilled from real discussions between frontline practitioners and FDEs; agents must elicit missing requirements from a simulated customer. Case‑specific evaluators compare the final artifacts against acceptance criteria, using customer‑approved FDE solutions as quality references.
Across 18 agent configurations, the highest mean delivery score reaches only 0.5510 (the FDE reference is 1.0), and the best delivery success rate (DSR) is merely 6.12%. These results indicate that reliable FDE‑level end‑to‑end delivery remains out of reach for the evaluated agents on underspecified real‑world business tasks.
Benchmark project page: https://baidubce.github.io/FM-Agent/FDE-Bench/
Review