Embodied agents must convert multimodal observations and user requests into executable actions while respecting product constraints, goal changes, and execution feedback. We introduce JoyInside-VLM-Agent, a ReAct‑style multimodal agent built via product‑conditioned trajectory construction and staged policy learning for robotic platforms and smart hardware.
A closed‑loop sandbox generates interaction trajectories that cover goal revisions and failure recovery, which are validated against execution outcomes and semantic criteria. The pipeline then applies supervised fine‑tuning and multi‑teacher on‑policy distillation, followed by reinforcement learning with DPER (Dual‑Source Privileged On‑Policy Self‑Distillation with Token‑Level Evidence Routing). DPER routes token‑level evidence between a frozen privileged teacher and a privileged student sharing rollout policy parameters, modulating reward‑derived token advantages while preserving their signs.
Training rewards combine generative process evaluation, multi‑judge rankings, and task‑queue consistency.
Results show JoyInside-VLM-Agent achieving a BFCL v4 overall score of 77.44, surpassing evaluated baselines; performance on other public benchmarks is mixed.
On the JoyInside‑Devices and JoyInside‑Robotics tracks, the system leads baselines on all reported metrics, thanks to domain‑specific skill refinement.
Two real‑world quadruped case studies illustrate visually grounded search, replanning after execution failure, and explicit failure reporting.
Review