Abstract
Vision-Language Navigation (VLN) requires an embodied agent to interpret a natural-language instruction and predict actions from temporally ordered visual observations. Adapting a multimodal large language model to VLN necessitates visual-language alignment, compact temporal inputs, action-space grounding, and stable training on the target hardware. This technical report presents PGN (Pangu Navigator), an offline VLN action-prediction system built on OpenPangu-7B.
Training Process
Training proceeds in two stages:
-
PGMM Alignment: A frozen EVA-ViT-G/14 vision encoder is aligned with a frozen language backbone by training a Q-Former and a two-layer MLP projector.
-
Expert Trajectory Adaptation: PGN adapts the aligned model to expert navigation trajectories using five-observation windows, epoch-dependent temporal sampling, and a reasoning-then-action output format; this stage freezes the aligned visual pathway and updates three structural-token embeddings and LoRA adapters.
Implementation Details
The implementation combines mixed-precision computation, selective FP32 computation, and DeepSpeed ZeRO-2 on eight Ascend 910B NPUs. Under teacher-forced, open-loop evaluation on 500 held-out expert trajectories, V9 reports a 62.29% Normalized Action Match (NAM) and a 100.00% Non-empty Rate (NER).
These metrics quantify offline expert-action alignment rather than closed-loop navigation success; evaluating error accumulation, path efficiency, and goal completion remains future work.
Blogger's Review: The PGN system showcases the potential of multimodal large language models in the field of visual navigation. With precise model alignment and innovative training methods, it significantly enhances action prediction accuracy. Future work should focus on integrating closed-loop navigation and real-time feedback mechanisms to further improve the system's practicality and intelligence.