Hybrid LLMs interleave a fixed‑size recurrent memory layer called Gated DeltaNet (GDN) with full‑attention layers whose key‑value (KV) cache grows linearly with context length. We first show that, in Qwen3.8-27B, the set of GDN heads that retain long‑term memories is determined by the model weights rather than by the input document. Next, we define the future utility of a KV entry as the change in attention output when the entry is removed, evaluated on the model’s own continuations. Empirically, the write gate, decay factor, and write magnitude emitted by GDN during the prefill stage already anticipate this utility before any query is observed. Because this relationship is distributed across many heads and layers, a linear map captures only the trend and misses the high‑utility tail, prompting us to introduce GateSight. GateSight equips each GDN head with a lightweight multilayer perceptron (MLP) that reads the three signals—without any attention scores—and learns a nonlinear mapping to predict utility. The predicted utility decides which KV entries each attention layer keeps; the actual amount retained per layer is governed by the model’s native attention output gate, whose mean varies by more than an order of magnitude across layers. At serving time, GateSight scores the cache once after prefill and physically discards entries deemed unnecessary. Experiments on SCBench and RULER‑4096 with Qwen3.8‑27B and, using the same recipe, the sparse‑attention model Qwen3.8‑Flash‑Next (up to the 262K‑token limit) show that with only one‑tenth of the attention KV cache, GateSight achieves the highest accuracy among five query‑agnostic methods on three of four model‑benchmark pairs and outperforms KVzip on all four, improving by 9.5–47.2 points. Notably, the overhead for full‑length context inference is negligible compared to prefill.
Review