Attention is a crucial component of transformers. However, the quadratic computational complexity of vanilla full attention concerning input size, along with the inability of linear attention variants to focus, presents challenges for computer vision tasks.
We provide a mathematical definition of generalized attention and formulate both vanilla softmax attention and linear attention within this general framework. We prove that generalized attention exhibits dispersion, meaning that as the number of keys approaches infinity, the query assigns equal weights to all keys.
Motivated by this dispersion property and the recent development of Mamba attention, we propose Scalable and Efficient Mamba-like Attention (SEMA), which utilizes token localization to avoid dispersion and maintain focus, complemented by theoretically consistent arithmetic averaging to capture global aspects of attention.
Our approach is validated on Imagenet-1k, where classification results demonstrate that SEMA is a scalable and effective alternative to linear attention, outperforming recent vision Mamba models on increasingly larger scales of images while maintaining similar model parameter sizes.
Blogger's Review: The introduction of SEMA not only showcases a significant improvement over traditional attention mechanisms but also offers a more efficient solution for computer vision tasks. By combining token localization with arithmetic averaging, SEMA achieves focused attention in processing large-scale data, making it a noteworthy area for further research and application.