Large language models (LLMs) are increasingly used as policies for autonomous decision‑making and planning across many domains. Despite strong reasoning abilities, LLMs struggle with long‑horizon tasks under partial observability. World models offer a way to boost policy performance during both training and inference. In inference, agents typically simulate the consequences of candidate actions with a world model before committing, which helps decision‑making. However, simulation alone does not capture uncertainty about the current state, a crucial factor in partially observable settings. To address this, we introduce Belief‑Based World Models (BB‑WMs) that maintain a belief distribution accessible to the LLM, indicating what is known and what remains uncertain. Before building methods to learn accurate BB‑WMs, we ask a fundamental question: does exposing a world model’s belief directly to an LLM policy improve decisions? Our experiments show that giving LLM agents access to BB‑WM beliefs enhances performance on partially observable tasks and complements existing simulation‑based world models. Code is released at https://github.com/skumar-ml/belief-world-models.
Review