We introduce VNT-PA (Visual Navigation Transformer with Pose Attention), a transformer planner that uses a set of depth keyframes indexed by camera pose as its context. Conventional learned navigation policies treat observations as a temporally ordered history, tying each observation to a timestamp via positional encodings, which hinders reuse of experience from earlier traversals. VNT-PA indexes keyframes by pose, and its attention operates on pose differences between keyframes rather than on their temporal order, enabling spatial rather than temporal context retrieval.
During training, VNT-PA imitates a shortest‑path planner that runs on the ground‑truth scene mesh. Given the current pose and the goal position, the model queries the pose‑indexed spatial context to predict the next action. On point‑goal navigation in HM3D validation scenes, VNT-PA achieves 93.3% success and 90.4% SPL (Success weighted by Path Length), outperforming baselines that encode the same context as a temporal sequence or treat pose merely as an input feature, both in navigation performance and training efficiency.
Because the spatial context is a pose‑stamped set, frames from different trajectories can be fused at test time, further improving robustness. Compared with conventional planners that rely on explicit maps, VNT-PA degrades more gracefully under localization noise. These results demonstrate that pose‑stamped experience can serve directly as the environment representation for a learned planner, and that making attention depend on pose differences rather than temporal order speeds up training and enhances long‑horizon navigation.
Review: VNT-PA cleverly embeds spatial pose information into the attention mechanism, breaking the limitations of temporal sequencing and offering a more efficient and robust learning framework for visual navigation.