LLM agents have long struggled to turn game knowledge into competent play, even when researchers augment the model with perception, memory, skill libraries, planners, or executable‑policy scaffolds. The rapid progress of coding agents raises two sharper questions: can frontier models now win games at all, and can they do so unaided, building the entire player themselves?
We introduce the Gauntlet framework, which follows a develop‑freeze‑evaluate pipeline to port games ranging from tiny arcade titles to full‑scale commercial releases into a single experimental setting. The core contract is deliberately minimal: a general‑purpose coding agent receives only a game description, a raw observation/action interface, and an empty policy file—no strategy, algorithm, or architecture is supplied.
During a single autonomous session the agent experiments with the live game and engineers a standalone controller. We then freeze the generated code and evaluate it on held‑out instances, with zero model calls during play. On an unpublished procedural roguelike, held‑out success rates vary from 0% to 86%, revealing a sharp generational threshold: every observed session of the newest‑generation system outperforms the best session of its predecessor.
At full‑game scale, a compiled raw‑API controller defeats every fair StarCraft II built‑in AI and two cheating variants, and single‑session programs achieve total conquest in Civilization (Freeciv) on unseen seeds. Although the win rates against novice AI are modest, this is the first instance of a language‑agent system winning complete games of this genre without per‑turn model calls and without a hand‑crafted tactical layer.
These findings indicate that frontier coding agents are beginning to capture long‑horizon strategy, and the frozen programs are inspectable. We term this capability compiled agency: development experience compiled into a persistent executable agent whose architecture is built entirely by the model.
Review