Offline goal‑conditioned reinforcement learning seeks to acquire universal skills by reaching any goal from any state without dense rewards. Model‑based approaches enable planning by scaling compute at test time, allowing complex tasks to be solved. Combining the two promises pre‑training on sub‑optimal demonstrations and zero‑shot adaptation via goal specification and planning. Existing methods, however, assume that effective planning requires accurate next‑state prediction in pixel or latent space. We show that value prediction is far more useful for planning, while precise state prediction is often unnecessary.
The key insight is that offline goal reaching naturally admits a closed‑form Monte Carlo return, which can replace temporal‑difference bootstrapping for value regression. We use this return as the sole supervision for latent world models: imagined latents are trained to predict goal‑reaching value rather than to match future states, implicitly preventing representation collapse. These value‑oriented representations allow imperfect world models to imagine on the level sets of the value field beyond the dataset transitions.
Based on this, we introduce VPFlow, a value‑predictive planner that couples the world model with a one‑step flow policy and performs planning in its noise space via evolutionary search. The method improves the policy with imagined generalized advantage and reduces out‑of‑distribution queries that are common in action‑space planning.
On 30 state‑ and pixel‑based OGBench datasets, VPFlow surpasses model‑free baselines by at least 30.3% and 10.1% in success rate, respectively, and outperforms latent planners LeWorldModel, PLDM, and TD‑MPC2. Remarkably, this is achieved despite a latent prediction error orders of magnitude larger than with state prediction, and further reducing this error actually lowers success rate. The results demonstrate that value‑predictive representations, not exact state prediction, are what offline planning truly requires.
Review: VPFlow’s value‑driven latent modeling breaks the reliance on precise state prediction, offering a more robust and efficient planning framework for offline goal‑reaching tasks.