ARC-Bench directly audits the ranking assumption of frozen JEPA‑style world models (JEPA‑WM). Reward‑free latent world models plan by scoring candidate actions with the distance between their predicted future embedding and the goal embedding; a smaller distance is taken as a better action. This implicitly assumes that latent closeness is rankable, i.e., ordering by latent distance agrees with ordering by true cost.
We introduce ARC‑Bench, a no‑leak, fixed‑candidate evaluation protocol that measures whether frozen JEPA‑WM can correctly rank candidate actions. The protocol is applied to publicly released JEPA‑WM checkpoints across navigation (PointMaze) and manipulation tasks.
Core Findings
- In the official manipulation audits, the top‑scored candidate is almost always sub‑optimal, showing a systematic inversion in ranking.
- The same inversion appears in maze domains.
- Replacing the visual backbone from DINOv2 with video‑pretrained V‑JEPA 1 and V‑JEPA 2 (ViT‑L / ViT‑G scale) does not eliminate the defect, indicating the issue is not tied to a specific backbone.
We further rule out trivial explanations such as data provenance, under‑training, matched‑budget backbone controls, and metric‑circularity controls.
Why the Defect Remains Hidden
Closed‑loop replanning evaluates actions at every step, which masks ranking errors made during the first plan. When we reduce the planner’s replanning frequency, success rates collapse in both navigation and manipulation domains. Episodes rescued by frequent replanning are enriched for severe first‑plan ranking failures in the PointMaze first‑plan diagnostic.
Thus, closed‑loop success rates systematically overstate the rankability of frozen latent representations. ARC‑Bench supplies a quantitative measurement of this flaw and explains the masking mechanism of closed‑loop replanning, providing an audit baseline for methods that adapt, amortize, or replan around latent‑space planners without directly checking JEPA‑WM action rankability.
Review: ARC‑Bench uncovers a fundamental limitation of current JEPA‑WM action ranking, urging researchers to validate the reliability of latent‑space rankings, especially under low replanning frequencies.