Visual navigation policies built on large pretrained models typically follow a common recipe: a dedicated visual encoder, a bespoke action head, and training on thousands of hours of cross-embodiment datasets. This paper introduces GemNav, a visual robot navigation policy that adapts a frozen Multimodal Large Language Model (MLLM) for short-to-medium horizon waypoint navigation using Low-Rank Adaptation (LoRA) solely on the language tower, without auxiliary visual encoders and continuous regression heads. Waypoints and categorical navigation signals share a single discrete token vocabulary generated by the language-model head, and a soft-decoded auxiliary loss recovers the metric structure discarded by pure cross-entropy training.
On a single 8.7-hour open corpus, roughly three orders of magnitude smaller than competing training sets, the policy transfers zero-shot to four physically distinct unseen environments, stopping within 0.25-0.42m of the goal across 20 real-world trials covering an open carpark, an obstacle carpark, a long outdoor chemical yard, and an indoor warehouse. Conditioning on short image histories improves offline metrics but yields no robot benefit, indicating a ceiling on what temporal context adds once pretrained vision features are in place. These results suggest that discrete-token adaptation of frozen MLLMs can provide a data-efficient, deployable alternative for foundation model robot navigation.
Blogger's Review: This study showcases the potential of multimodal large language models in robot navigation, particularly through the adaptation of discrete tokens, significantly reducing reliance on large datasets. It offers a fresh perspective for future robotic navigation systems, especially in data-scarce environments, demonstrating impressive transfer capabilities and efficiency. Overall, GemNav could represent a disruptive advancement in the field.