In partially observable environments, agents must act without full knowledge of the world state and therefore rely on uncertain state‑estimation pipelines. Obtaining grounded and verifiable symbolic plans under such uncertainty remains a fundamental challenge. Recent work has leveraged Vision‑Language Models (VLMs) to connect perception with symbolic reasoning, following two main paradigms. The first, VLM‑as‑planner, maps images directly to action sequences. The second, VLM‑as‑grounder, grounds observations into symbolic predicates that serve as the initial state for off‑the‑shelf planners. Both approaches ignore uncertainty during planning, compromising robustness. We introduce a third paradigm—VLM‑as‑probabilistic‑grounder—that treats VLM predicate groundings as a probability distribution over symbolic states. This enables belief‑space planning and the generation of robust plans under uncertainty. Experiments in simulated household‑robot settings show that, compared with deterministic grounding, the probabilistic approach markedly improves task success and robustness, highlighting how foundation models can support reliable planning in uncertain domains.
Review