[CS.AI] LISA: Efficient Long-Context Reasoning with Linear-Indexed Sparse Attention

Abstract

Recent advances in long chain-of-thought reasoning models, such as DeepSeek-R1, have led to increasingly longer inference context lengths under the test-time scaling paradigm. However, the O(n^2) computational complexity of standard self-attention causes inference costs to grow sharply with long sequences, limiting the deployment of long-CoT reasoning in production settings. To address this, we propose LISA (Linear-Indexed Sparse Attention), a plug-and-play attention replacement module that requires no pretraining from scratch.

LISA integrates two lightweight components in parallel within the original model:

1. Linear Attention module that provides long-range memory with O(n) time complexity;
2. Lightning Indexer that selects the top-M important tokens from the full context to feed into a Sparse Self-Attention.

The two branches are fused via a gating mechanism, reducing inference complexity from O(n^2) to O(nM).

Code Implementation Example

Here is a simplified code implementation of the LISA structure:

class LISA {
public:
    void forward(vector<Token> context) {
        auto longRangeMemory = linearAttention(context);
        auto importantTokens = lightningIndexer(context);
        auto sparseAttentionOutput = sparseSelfAttention(importantTokens);
        auto finalOutput = gatingMechanism(longRangeMemory, sparseAttentionOutput);
    }
};

With this design, LISA efficiently handles longer contexts while significantly enhancing the potential of long chain reasoning applications.

Blogger's Review: The introduction of LISA effectively alleviates the computational bottleneck in long sequence reasoning, showcasing the advantages of combining sparse attention with linear attention. This innovative approach opens new possibilities for the practical application of long chain reasoning models, warranting further attention and research.