Learning robot manipulation policies typically requires large amounts of demonstration data, which are expensive to collect on real robots. Recent approaches generate robot demonstrations by recovering motion from human videos and validating the resulting trajectories in simulation. However, methods that focus solely on motion‑reference adaptation retain the demonstrated contact strategies and sub‑task orders, limiting behavioral diversity, while insufficient understanding of task requirements and scene relations can produce invalid candidates and reduce generation efficiency.\
To overcome these limitations, we propose KnowDemo, a framework that leverages structured manipulation knowledge extracted from human videos to generate diverse robot demonstrations for a target workspace. We first develop a knowledge extraction and reasoning module based on a vision‑language model (VLM) that maps object and action descriptions to inferred task conditions, demonstration references, and permissible execution variations, thereby separating task requirements from demonstration‑specific choices.\
The extracted knowledge is then resolved against entities and geometry in the target scene to guide candidate generation and screening, followed by motion planning and simulation validation. The resulting demonstrations exhibit multimodal behavior, including alternative contact strategies and valid sub‑task orders, and are annotated with structured execution labels. Experiments show that, beyond a reference‑only configuration, KnowDemo discovers additional verified execution modes and improves candidate planning success through task‑guided grasp sampling.\
To validate the usefulness of the generated data for policy learning, we fine‑tune the pretrained $\pi_{0.5}$ model on simulation data and achieve sim‑to‑real transfer across three tasks. Project page: https://zhiyuan-gao.github.io/knowdemo/
Review