World models promise a general route to embodied intelligence: learn predictive dynamics once, then reason, plan, and act with them. Recently, the representations underneath such models are often pretrained on massive video, interaction, and multimodal corpora, raising a question that prediction quality alone cannot answer: when is a learned representation truly actionable? We identify a failure mode called counterfactual collapse: the model generates visually plausible futures but fails to distinguish interventions that lead to different behavioral outcomes. This arises because the representation is optimized for perceptual similarity rather than intervention structure—the very objective of most large‑scale pretrained encoders. To address this, we introduce Counterfactual Latent World Models (CLWM), which consist of:
- a recurrent belief‑state encoder that aggregates past observations into a latent belief;
- action‑conditioned latent dynamics that predict latent transitions given an action;
- a contrastive counterfactual objective that forces futures induced by distinct interventions to remain separable in latent space, even when their observations look alike.
Across three benchmarks—occluded manipulation, aliased navigation, and long‑horizon manipulation—CLWM markedly improves planning success. For instance, on the Occluded Push task success rises from $65.1\% \to 74.6\%$, on the Aliased Maze from $67.3\% \to 78.9\%$, and exploitative planning failures on Deferred Kitchen drop from $18.4\% \to 9.7\%$. Ablations attribute most gains to hard counterfactual negatives, especially perceptual‑alias negatives. We also propose a counterfactual separability metric that tracks planning success across five baseline model classes ($r \ge 0.94$) and is representation‑agnostic: given intervention‑outcome labels, it can audit any encoder—pretrained or trained from scratch—before a planner trusts it. We have not yet measured it on large‑scale pretrained encoders; here we establish the metric and its relationship to planning success for world models trained from scratch.
Review