Post‑training is crucial for enhancing the reasoning ability and task‑specific expertise of large language models (LLMs). Existing methods such as Group Relative Policy Optimization (GRPO) suffer from training instability in practice because they rely on importance sampling. We introduce Group Variance Policy Optimization (GVPO), which incorporates the analytical solution of KL‑constrained reward maximization into its gradient weighting scheme. This scheme can be interpreted as the mean‑squared error between the central distance of implicit rewards and that of actual rewards.
GVPO offers two key advantages:
- The analytical solution guarantees a unique optimum that exactly matches the KL‑constrained reward maximization objective;
- It allows flexible sampling distributions without requiring importance sampling.
Formally, KL‑constrained reward maximization is expressed as @@@MATH_BLOCK0@@@\nabla J = \mathbb{E}{\pi_{old}}\big[ w\,(\hat{R}-R)^2 \big]$$, which is the mean‑squared error of the central distances.
Beyond general post‑training, GVPO naturally extends to On‑Policy Distillation (OPD). In the OPD setting, GVPO still provides a unique optimum and supports a broad family of extended OPD objectives, offering a principled foundation for objective design.
In summary, GVPO unifies theoretical guarantees with practical adaptability, establishing a new paradigm for reliable LLM post‑training and versatile on‑policy distillation.
Review