State‑of‑the‑art large language model (LLM) agents have demonstrated the ability to solve increasingly complex tasks, many of which have measurable human solutions. This raises a fundamental question: can LLM agents go beyond what humans have already solved? When well‑trodden human solutions become insufficient due to lack of context or training data, the capacity to devise sophisticated, self‑generated strategies becomes crucial. We investigate this capability through the communal practice of video‑game speedrunning. In speedrunning, participants compete to complete a game as quickly as possible under specific constraints, often uncovering unconventional play styles that require deep mastery of the underlying mechanics. To this end, we introduce SPEEDRUNBENCH, a benchmark spanning nine different games. Success on this benchmark demands that agents repeatedly refine their strategies, reflect on performance, exploit accumulated knowledge, and reason over long‑horizon action sequences to outpace both themselves and others.
Our experiments reveal that frontier LLM agents can approach human world records on simple platformers, yet they lag behind human performance on longer, more intricate games when constrained by practical compute budgets. These findings suggest that SPEEDRUNBENCH serves as a valuable testbed for studying agents’ strategy‑formation abilities and provides a saturation‑resistant evaluation metric, as there is almost always a faster completion time waiting to be discovered.
Review