Abstract
The recent emergence of vibe-coding workflows is changing the expectations placed on coding agents. Instead of merely completing code under fully specified instructions, agents are increasingly required to transform incomplete product intent into working software by combining various abilities, including planning, requirement clarification, tool use, debugging, and repository-level construction.
However, existing benchmarks have not fully adapted to this shift, evaluating agents on static, fully specified tasks. In this paper, we introduce ICAE-Bench, a benchmark for evaluating coding agents in interactive project-building settings. The core idea is to start from a fuzzy product requirement, simulating the dynamic paradigm with an automated User Agent.
To make this setting both realistic and evaluable, ICAE-Bench incorporates three key designs. First, to avoid ambiguity from unconstrained fuzzy requirements, each task derives its ambiguity from a precise real open-source repository with executable behavior. Second, to ensure high-quality and reproducible user simulation, ICAE-Bench grounds interactions through User Agent Data, allowing the User Agent to reveal hidden constraints without inventing new requirements or leaking implementation artifacts. Third, to fairly evaluate open-ended repositories, ICAE-Bench employs standardized black-box tests alongside multi-dimensional diagnostics, including functional correctness, semantic and API similarity, structural fidelity, design quality, and interaction quality.
Blogger's Review: The introduction of ICAE-Bench addresses the practical needs of modern coding agents, particularly the complexities and uncertainties in dynamic project building. This marks a significant advancement in evaluation methods, as it effectively assesses agents' performances in real environments through precise open-source repositories and high-quality user simulations, driving the development of coding agents forward.