World models enable observers to explore and interact within generated environments. Recent studies show that when an observer revisits a region, current models often fail to regenerate it consistently. We treat this as a context retrieval problem. LLM agents manage context retrieval at two stack layers: the model layer’s KV cache determines how much context fits in memory, while the agent harness (e.g., Claude Code) selects the context sent to the LLM. Prior work has mainly focused on the model layer.
This paper makes three contributions. First, we build a benchmark to measure context retrieval in world models. The benchmark generates a fresh world each episode, preventing models from recalling rooms via training data, and separates true context retrieval from guesses based on nearby frames. Second, we implement an agent‑level harness that performs context compaction for world models—a previously unexplored area. Third, we use the benchmark to evaluate four world‑model designs.
Results show that context compaction allows models to retrieve rooms up to four times beyond their original context window without retraining. However, models that rely on sliding‑window attention lose most of their retrieval capability during dreaming. Consequently, a dreaming world agent benefits more from extending its model’s reach via context compaction than from switching to sliding‑window attention at the model layer.
Review