Abstract
This paper investigates the attribution question of performance in ARC-AGI-3 agents by designing four Codex-based agents:
- Textual baseline
- Flexible-interface executable world model (without replay verification)
- Executable model with scheduled simplification
- Fixed-interface verification treatment (retaining simplification and requiring exact reproduction of recorded observations)
The main study evaluates all four agents under high and xhigh reasoning efforts on public ARC-AGI-3 games. Exploratory follow-ups assess the textual and verification variants at xhigh and max reasoning efforts.
The most robust finding is that every agent variant improves with stronger models and greater reasoning efforts. Within each model-effort setting, differences among variants are smaller than anticipated, while the effects of individual components vary across settings. Requiring a persistent executable deliverable is not universally beneficial: the textual variant outperforms the flexible-interface executable variant in both gpt-5.5 settings. Simplification improves performance in three of the four model-effort settings, with the weakest setting as the only exception. The complete verification treatment ranks first in all four settings, although it uses substantially more resources. In the gpt-5.6-sol follow-up, the verification variant fully solves every public game at both reasoning efforts, achieving about 99% RHAE and using fewer than half the total actions of the human baseline. Since the model postdates these games and held-out performance remains untested, this result should be interpreted as saturation of the public set only.
Blogger's Review: This paper offers valuable insights into the relative importance of executable world models and verification mechanisms in tackling complex tasks, providing profound implications for future AGI research. The impact of model performance, particularly, warrants further exploration and experimentation.