We introduce SynthDemo‑RL, a teacher‑student framework designed to overcome the exploration challenge of sparse binary rewards in fine‑tuning Vision‑Language‑Action (VLA) models. The teacher automatically converts privileged simulator states into successful manipulation trajectories; the student is distilled from these synthetic demonstrations via supervised fine‑tuning (SFT), and then refined with PPO using binary task‑success rewards.
To complement average success rate, we propose the metric reward coverage—the fraction of tasks that exhibit at least one successful trajectory under a fixed evaluation protocol. On LIBERO‑PRO, a public benchmark of 57 perturbed LIBERO tasks with no demonstrations, a pi_0.5 policy fine‑tuned on the original LIBERO tasks achieves 0% success on 27 tasks. Direct PPO with the same recipe and compute as SynthDemo‑RL’s refinement rescues 10 of these tasks, leaving 17 at 0%.
SynthDemo‑RL generates 50 synthetic trajectories per task without any new human demonstrations, rescuing all 27 tasks and reaching average success rates of 97.8% (Position) and 97.1% (Task) on LIBERO‑PRO. The same pipeline on standard LIBERO attains 96.0% success, within 1.7 points of a pi_0.5 policy trained on 50 human demonstrations per task.
We further validate the pipeline on RoboTwin 2.0, confirming that trajectories from a policy trained in a MuJoCo twin execute open‑loop on a physical robot.
Review