Large language model (LLM) agents can generate fluent action sequences across many tasks, yet they break down when the environment is only partially observable. Ambiguous feedback pushes the model into premature commitments, a single informative observation can collapse uncertainty onto a wrong hypothesis, and policies drift as history grows. We trace these failures to a common structural flaw: the deployed LLM agent is a history‑conditioned policy that lacks an explicit belief over hidden states.
To address this, we introduce the Belief‑State Engine (BSE), an inference module placed outside the LLM. BSE maintains a Bayesian posterior over the latent states of a given POMDP (Partially Observable Markov Decision Process) and, at each decision step, exposes only this posterior to the LLM while hiding the raw action‑observation log.
BSE must satisfy four axioms: (1) the posterior is a probability distribution over all possible hidden states; (2) updates follow Bayes' rule; (3) the posterior depends solely on the previous posterior, the latest observation, and the chosen action; (4) when the posterior converges it corresponds to a unique most‑likely hidden state. An internal state meeting these conditions is called belief‑consistent.
Theoretically, we prove that if the LLM never sees the raw history, the LLM‑BSE pair constitutes a sound Markov policy on the belief MDP induced by the underlying POMDP, thereby inheriting Bellman optimality guarantees from classical POMDP theory.
We evaluate the architecture on the classic Tiger POMDP and a red‑team attack‑graph task, comparing against six baselines: a reactive LLM, Chain‑of‑Thought, ReAct, a natural‑language belief tracker, QMDP, and POMCP. Across both domains the BSE‑augmented agent improves task return, belief calibration, and decision consistency. Ten targeted ablations that remove each architectural component confirm that the gains are not tied to any specific model.
Code, environment specifications, prompt templates, and seed logs are released alongside the paper for reproducibility.
Review: BSE equips LLMs with a principled probabilistic belief layer for partially observable settings, effectively transplanting the theoretical strengths of POMDPs into natural‑language interaction and highlighting the synergy between structured inference and large‑scale generation.