In a Partially Observable Markov Decision Process (POMDP), a belief is the posterior distribution over the hidden state conditioned on the agent’s history of actions and observations. It encapsulates everything the history says about the state, so planning in a POMDP is equivalent to planning in a Markov Decision Process (MDP) over beliefs. Even for finite‑state POMDPs, the belief MDP has a continuous state space, making exact planning intractable in general. Consequently, planners must approximate beliefs and estimate values. Monte Carlo planners maintain an approximate belief at every node of a search tree and a value estimate at each leaf. Typical approximations consist of a set of sampled states; leaf values are obtained from rollouts, hand‑crafted bounds, or learned networks. Policies and values derived from these approximations are limited by their quality, and prior methods that use learned networks rely on hand‑chosen summaries of the sampled states. An ideal belief representation should be computable from the parent node’s representation via the POMDP dynamics, allow the search to sample from it, and be learnable so that it provides the information needed by policies and value functions. We introduce Belief Embedding Tree Search (BETS), a Monte Carlo planner that satisfies these requirements. BETS represents each node’s belief with a Generative Distribution Embedding (GDE) and updates it through the POMDP dynamics during search. The belief model is trained jointly with policy and value networks using interaction experience, with the realized hidden state of the POMDP serving as the training target for the belief model. We evaluate BETS on six configurations of RockSample, LightDark, and PocMan against BetaZero and three planners without learned components. BETS is competitive or leading on all six tasks, while each baseline falls far behind on at least one. In high‑dimensional belief settings, BETS outperforms the baselines by a large margin, even without any search.
Review: BETS leverages a learnable belief embedding within the search tree to achieve efficient approximation of high‑dimensional beliefs and accurate value estimation, highlighting the strong potential of generative models for POMDP planning.