NeFut Logo NeFut
中 Admin Login

[CS.AI] Disentangling Multimodal Rotary Positional Embedding in Vision-Language Models

Published at: 2026-11-11 22:00 Last updated: 2026-11-12 00:51
#AI #Machine Learning #Neural

Vision‑language models commonly adopt Multimodal Rotary Positional Embedding (MRoPE), an extension of RoPE that adds temporal, height, and width axes to encode the spatial structure of visual tokens. MRoPE also assigns height and width coordinates to text tokens, which lack inherent spatial layout, thereby injecting spatial positional effects into text‑vision attention. Existing MRoPE variants explore different frequency allocations and coordinate layouts, reflecting two intuitions: the cross‑modal spatial effect might serve as a useful reference for linking language to visual regions, or it could introduce unnecessary positional bias that hampers text‑vision interaction.\nWe first disentangle intra‑visual spatial structure from cross‑modal spatial effects and identify a shared‑coordinate‑layout constraint that prevents eliminating cross‑modal effects while preserving intra‑visual encoding. Guided by this insight, we construct controlled experiments to isolate each contribution, testing across multiple model scales, MRoPE variants, and benchmark tasks.\nThe findings show that cross‑modal spatial effects provide little extra benefit as spatial references and can even degrade performance, whereas intra‑visual spatial encoding remains valuable for fine‑grained grounding.\nReview: The spatial bias introduced by cross‑modal positional embeddings is largely unnecessary; future work should prioritize strengthening visual‑internal spatial representations rather than indiscriminately adding spatial coordinates to text tokens.

Original Source: https://arxiv.org/abs/2611.13334

[h] Back to Home