As large language models (LLMs) become more capable of writing code, the main bottleneck shifts to whether they truly grasp what users want. Users often start with vague intent, leaving critical requirements to be clarified through dialogue. In repository‑level development, misunderstandings can propagate through the code base and cause expensive rework. Existing benchmarks usually evaluate implementation from complete specs or underspecified requests, ignoring the pre‑coding clarification step; prior clarification studies focus on function‑level tasks and do not reveal performance on larger projects. To address this gap we introduce ReqClarifyBench, an interactive requirement‑clarification benchmark for repository‑level software development. The benchmark comprises 341 tasks across game, mini‑app, and web development, each containing a vague user intent, a human‑validated requirement document, and code evaluation artifacts. We employ LLM‑based user simulators that emulate diverse personas drawn from real conversations. Six mainstream LLMs are evaluated on clarification effectiveness and the quality of the generated requirement documents. Results show that stronger clarification does not necessarily lead to higher document quality, and performance varies markedly across personas. Moreover, adding a coding‑agent layer does not improve requirement document quality compared with the base LLMs. These findings offer valuable insights for building end‑to‑end LLM‑driven software development systems.
Review