In the field of video understanding (classification), we present a split space-time attention model, VideoSEMA, which consists of a scalable and efficient Mamba-like attention (SEMA) block and a softmax temporal attention. In each frame, SEMA attention applies local window attention in parallel with global averaging under a Mamba macro-architecture, referred to as Mamba-like. Under certain rank conditions, we prove that the computationally cheaper split space-time attention is equivalent to full space-time attention.
On the benchmark K400 dataset, VideoSEMA outperforms heavier vision transformers and Mamba models. On the benchmark SSv2 dataset, VideoSEMA achieves top-1 accuracy among models of similar parameter sizes. As image resolution scales up from the standard $224^2$ to $1024^2$ on K400, VideoSEMA degrades much more gracefully in accuracy than VideoMamba. It is promising to extend VideoSEMA to longer videos with dilated/sparse temporal attention.
Blogger's Review: VideoSEMA significantly enhances the efficiency and accuracy of video understanding, particularly excelling in high-resolution scenarios compared to existing models. This advancement offers new perspectives for handling longer videos, making it a noteworthy development to watch.