End-to-end autonomous driving has achieved state-of-the-art performance on benchmarks and real-world deployments. However, its standard training recipe is expensive across all stages: collecting and labeling millions of driving frames is costly, and closed-loop RL on images is bottlenecked by the per-step cost of photorealistic rendering plus a forward pass through a large vision backbone. Self-play in vectorized simulators changes the economics: millions of rollout steps per second, with a state distribution naturally rich in collisions, near-misses, and recoveries that no driving log contains.
Our approach exploits this asymmetry by decoupling learning to drive from learning to see. We pretrain a single policy by self-play, then align its latent space with a pretrained vision backbone through action KL divergence and a batch-relational low-rank structural loss. The action target comes from the self-play policy, so alignment never supervises against a logged trajectory: a paired dataset of (image, scene-state) frames suffices, with no need for the curated expert demonstrations that imitation pretraining is built on.
On photorealistic 3D Gaussian splatting closed-loop scenarios, the resulting end-to-end policy matches or exceeds prior end-to-end methods.
Blogger's Review: TerraTransfer significantly reduces the training costs associated with traditional autonomous driving models through a unique self-play strategy, enhancing efficiency and demonstrating how high-performance driving policy learning can be achieved without expert demonstrations. This innovation presents new perspectives and methods for the future development of autonomous driving technology.