Coding agents can modify and test code across large software projects, and game development is a domain where they must implement gameplay rules. A game may reach a valid end state even after violating its rules during execution. Existing game‑development benchmarks either replay fixed examples, score videos, or ask another model to judge outcomes, but none check rules at every simulation tick nor guarantee fully reproducible verdicts. We therefore introduce GameLogicBench, a benchmark comprising 72 gameplay‑logic tasks built on Godot projects. An automated evaluator checks each game's rules at every tick, covering 403 hand‑designed scenarios; parameter variations yield 1,451 test cases. The evaluator must accept all correct implementations while rejecting mutants that lack a required capability, ensuring it measures behavior rather than implementation choice. Tasks range from isolated mechanics to multi‑system interactions and repository‑scale features. Across 20 language‑model and scaffold combinations, the best run solves 52.78% of tasks. Under Claude Code, all twelve models solve fewer tasks as scope expands from isolated mechanics through interacting systems to repository‑scale features. Agents inspect code more often and make more tool calls on repository‑scale tasks than on isolated‑mechanic tasks. Most failed submissions are runnable but implement some required game behavior incorrectly. We compared evaluator versions with and without mutant validation and found that without validation, incorrect submissions passed. A separate analysis shows agents copy code from public repositories when network access is open. Thus reliable evaluation depends both on what the tests reject and on the external code agents can access.
Review