Autonomous software generation (ASG) aims to turn human requirements directly into executable applications, yet delivering an application does not guarantee that its interacting components satisfy the specified behavioral requirements. To address this, we introduce GameASG-Bench, which incorporates behavioral testability into the generation task for game development.
Before generation, an evaluation interface specification is declared, fixing legal starting scenarios, player‑level actions, stable snapshots, rejection behavior, and invariants while keeping private implementations open. The implementation consists of two kinds of checks:
- L1 static checks that assess source‑level compliance;
- L2 browser‑executed checks that combine semantic observations, real input, and runtime evidence.
We realize this protocol as 47 browser‑native game‑generation tasks covering 12 primary genres and both 2D and 3D interaction, each equipped with executable checks and an independently verified reference implementation.
Our experiments answer four key questions: end‑to‑end agent performance, tool access and nominal turn budget, reasoning effort, and harness choice. Across nine agent stacks, the highest observed mean L2 check pass rate is 93.2%, yet the highest strict task success rate—requiring all L1 and applicable L2 prerequisite and core requirement checks—is only 55.3% (26/47 tasks). For DeepSeek‑V4‑Flash, full tool access and larger nominal turn budgets yield more strict task successes, but the success rate is not monotonic in reasoning effort. Both tested harnesses achieve 18 strict task successes, with only ten tasks succeeding under both.
These findings expose task‑level compliance gaps that high average check pass rates tend to obscure.
Review