Recent work shows that pre‑trained, instruction‑tuned vision‑language models (VLMs) can map instructions and observations to high‑level plans, yet they struggle to turn those plans into reliable, low‑latency action sequences in unfamiliar settings. In contrast, world‑model controllers excel at fast observation‑to‑action mapping but lack open‑ended task guidance. To bridge this gap, we introduce Instruct‑to‑Act, a system that trains a world‑model controller to act autonomously at high frequency when conditioned on sparse, higher‑latency textual instructions generated by a VLM planner.
To make the controller language‑instructable, we relabel segments of controller policy rollouts with synthetic instructions and jointly optimize a behavior‑cloning loss together with existing reward‑maximization and world‑modeling objectives. We evaluate the approach across seven embodied environments, including three multi‑agent scenarios where VLM planners coordinate via language while the trained controllers serve as actuators. Under matched observation and action spaces, the decoupled method consistently outperforms controller‑only and direct VLM‑action baselines, preserves fast control, and allows swapping different pretrained VLM planners without fine‑tuning. Overall, our method remains competitive with strong vision‑language‑action and multi‑agent RL baselines on six of the seven tasks.
Blogger's Review: The paper presents a clever separation of high‑level language planning from low‑level real‑time control, preserving the open‑ended reasoning of VLMs while leveraging the responsiveness of world‑model controllers. It offers a promising pathway toward truly instruction‑driven embodied agents in complex environments.