We introduce SpireBench, a benchmark that lets large language models play the full commercial deck‑builder Slay the Spire (StS). Early decisions in the game shape long‑term success, making it an ideal testbed for long‑horizon strategic reasoning. We evaluated 20 LLMs at the game’s highest difficulty. Frontier models outperform heuristic baselines and match the average human player, yet they remain far behind expert players. The performance gap widens as the run lengthens: the strongest models win the first act as often as experts but fall behind thereafter.
The key differentiator between strong and weak models is how well they handle each individual fight. All models tend to misjudge risk, preserving health instead of spending it to build a stronger deck, which harms overall run performance. A comprehensive, self‑consistent renaming of every game term reveals that part of the success stems from prior knowledge of the game rather than learning from memories written across runs. Those memories are often incorrect and overly cautious, providing little improvement in subsequent runs.
Our harness is the first to give a model access to the complete original game, and we compare it with existing, simplified harnesses. Overall, SpireBench offers an unsaturated long‑horizon benchmark equipped with tools to pinpoint where and why agents fail.
Review