Vision-language models (VLMs) struggle with generalization in interactive physical reasoning, particularly in unseen tasks and environments. Two key failure modes are evident: hallucinated chain-of-thought (CoT) reasoning contradicting physical reality, and misalignment between the model's reasoning and actions. We present VAORA (Visual Action Outcome Reasoning Alignment), a novel reward design addressing both issues.
VAORA introduces two complementary rewards:
- Visual Alignment Reward: anchors VLM reasoning to the visual context independent of the agent's action.
- Visual-Action Alignment Reward: grounds reasoning in the visual outcome induced by the model's action.
Together, these rewards suppress hallucinated CoT and reduce the gap between reasoning and behavior. To enhance training stability, we employ smooth, dense rewards by estimating success probabilities using a pre-trained in-domain expert agent.
Experiments on PHYRE and Virtual Tool confirm our performance across novel-task and unseen-environment settings, validating that grounded and generalizable physical intelligence can be induced through VAORA.
Blogger's Review: The introduction of VAORA marks a significant step forward for VLMs in physical reasoning. By integrating visual and action outcome alignments, it effectively addresses the challenges of model generalization. This method merits further exploration and application in diverse domains.