NeFut Logo NeFut
中 Admin Login

[CS.AI] DAMI: Dual-Axis Memory Initialization for Enhanced Long-Context Retrieval in Gated Linear Attention

Published at: 2026-11-04 22:00 Last updated: 2026-11-05 00:45
#algorithm #AI #Machine Learning

Linear attention scales linearly with sequence length, yet its finite‑state memory can restrict long‑context retrieval. In gated linear recurrences, memory retention depends on state decay, which is jointly shaped by gate parameters $A_{\log}$, $dt_{bias}$ and input activations. Controlled Gated DeltaNet (GDN) experiments show that these parameters remain highly correlated with their initial values after training, highlighting the role of initialization.\nWe therefore introduce Dual‑Axis Memory Initialization (DAMI), which configures these parameters jointly before training. DAMI assigns geometric timescales across heads to cover varied retrieval distances and adopts a low‑high‑low input‑sensitivity profile across layers. A small activation sample calibrates the GDN‑family implementation to meet both behavioral targets without adding trainable parameters.\nAcross GDN, KDA, GDN2, Mamba‑2 and RWKV‑7, DAMI improves average RULER‑style retrieval accuracy by 5.29–16.71 percentage points over random‑initialization baselines. Performance on language modeling, commonsense reasoning and a 2K real‑world retrieval benchmark remains broadly comparable to those baselines.\nCode is available at https://speedai.com/research/dami\n\nReview

Original Source: https://arxiv.org/abs/2610.41994

[h] Back to Home