Efficient attention mechanisms such as linear and sliding‑window attention reduce Transformer complexity from $O(N^2)$ to $O(N)$, yet their weakened in‑context recall often forces the insertion of full‑attention layers, leaving the quadratic bottleneck unresolved for long‑context inference. Orthros attention addresses this by shifting the efficiency focus from structural allocation to phase‑aware decoupling. As a unified module, Orthros dynamically switches its computation pattern across processing phases, enabling linear‑complexity prefilling and full‑attention decoding. This yields a lower asymptotic complexity than vanilla Transformers in typical scenarios where prefilling dominates decoding. Theoretical analysis shows Orthros’s overall cost approximates $O(N_{prefill}+N_{decode}^2)$ when $N_{prefill}\gg N_{decode}$, compared with $O((N_{prefill}+N_{decode})^2)$ for standard Transformers. Empirically, Orthros demonstrates: (1) strong language‑modeling scalability across model sizes and competitive performance against diverse architectures; (2) a data‑efficient Transformer‑to‑Orthros conversion that can surpass the original model; (3) an end‑to‑end inference speedup of $21.9\times$ for prefilling while preserving long‑context abilities, evidenced by near‑perfect 128K needle‑in‑a‑haystack recall and Transformer‑level results on real‑world in‑context tasks. These combined gains in performance, flexibility, and efficiency position Orthros as a promising successor to Transformers in industrial applications.
Review