Multimodal large language models (MLLMs) struggle with spatial reasoning, particularly when it requires perspective transformation. They often rely on camera-centric cues rather than reasoning from the reference object's viewpoint, leading to systematic errors in non-camera reference settings.
This paper first analyzes this failure mode and identifies object orientation as a key factor causing camera-centric shortcut behavior. To address this, we propose OrientSAM, an orientation-aware spatial alignment framework for multimodal models.
OrientSAM injects explicit orientation information into multimodal representations via orientation-aware tokens and Fourier-based angle encoding, and adopts a curriculum learning strategy to progressively enhance perspective-aware reasoning. Furthermore, we build a spatial data construction pipeline to generate orientation-aware spatial supervision from large-scale images.
Experiments on datasets like Spatial-MM, ViewSpatial, and 3DSRBench show that OrientSAM consistently outperforms strong baselines, particularly on non-camera-view, person-centric, and orientation-sensitive tasks. The results further demonstrate that explicit orientation modeling is crucial for mitigating camera-centric shortcut behavior and enabling more robust allocentric spatial reasoning in multimodal models.
Blogger's Review: OrientSAM effectively enhances the accuracy of multimodal spatial reasoning by incorporating orientation awareness, highlighting the significance of directional information in model inference. Its outstanding performance in non-camera perspective tasks suggests promising applications for future research in multimodal understanding.