We introduce HyScale, a training system designed for learned sparse attention over million‑token contexts. Learned sparse attention has become common in production since DeepSeek‑V3.2, yet the selection step now dominates training cost. For each query the model scores every previous token while only $k$ tokens are attended; current pipelines materialize a full score matrix and repeat the selection during recomputation. The key insight is that selection outputs a set of indices rather than activations. Building on this, HyScale adds two mechanisms: HyTopK and selection replay. HyTopK computes the exact top‑$k$ for each query on‑the‑fly without storing a score table, and selection replay reuses the forward‑pass indices during the backward pass. Coupled with holistic system optimizations, HyScale speeds up GLM‑5.1 training on 1M‑token sequences by 2.07$\times$ over Megatron‑LM on 512 H20 GPUs, and accelerates the CSA layers of DeepSeek‑V4‑Flash and the QSA layers of Qwen3.8‑Flash‑Next by 2.01$\times$ and 1.57$\times$ respectively on 32 H800 GPUs.
Review: By treating selection as an index operation and replaying it in the backward pass, HyScale cuts both memory and compute overhead, making long‑context sparse‑attention models far more practical for large‑scale deployment.