Reinforcement learning with verifiable rewards (RLVR) has markedly improved the reasoning ability of large vision‑language models (LVLMs). However, standard on‑policy RLVR algorithms encounter a critical bottleneck when trying to preserve and reinforce visually‑grounded reasoning: valuable visual reasoning trajectories are discarded after a single update, and a uniform token advantage allocation prevents the model from strengthening essential perception or reasoning steps.
To address this gap, we introduce PIVOT, a dual‑level learning framework that anchors policy optimization around informative visual reasoning signals. PIVOT first adds a self‑calibrated experience replay mechanism that selectively gathers and replays historically grounded visual experiences, using them as stable reference anchors for policy updates. Building on this, we design a vision‑guided advantage allocation mechanism that assigns extra vision‑aware advantages to tokens based on their local visual support and impact on downstream reasoning, allowing the model to receive stronger reinforcement on critical visual steps.
Extensive experiments on diverse benchmarks demonstrate that PIVOT achieves highly competitive performance in enhancing LVLM multimodal reasoning, often matching or surpassing state‑of‑the‑art methods.
Review: By combining experience replay with vision‑aware advantage allocation, PIVOT mitigates the information loss inherent in online RLVR and offers a more robust learning trajectory for visually‑grounded multimodal reasoning.