Attention-based KV cache eviction (H2O and its descendants) compresses the memory-constrained state of a long-context model by ranking tokens based on accumulated attention mass, treated as signal energy, and retaining those with the highest energy. However, in schema-dense input streams like nested JSON, this score acts as a non-stationary filter that disproportionately retains noise: non-content roles (such as delimiters or whitespace) carry an order of magnitude more energy than any content role, and structural KEY tokens are over-retained at approximately 1.8 times the rate of answer-carrying VALUE tokens, collapsing exact-match accuracy from 88% to 0% at a 5% budget as the signal-to-noise ratio of the retained state degrades.
A counterfactual experiment establishes that suppressing KEY tokens is the best deployable filter. Our retraining-free, role-conditional allocation over SnapKV's windowed score, governed by a single tuned hyperparameter, closes 63-98% of the H2O gap at sub-20% budgets and modestly matches or exceeds full-cache accuracy at higher budgets. This small, seed-sensitive denoising effect is borderline significant at B=0.50 and indistinguishable from zero at B=0.30 over four seeds. A 15 MB linear role probe supplies these labels at negligible inference cost, though matching parser-level downstream accuracy remains an open question.
Blogger's Review: This paper presents an effective solution for correcting structural-role bias in LLM inference through adaptive filtering mechanisms, significantly enhancing model accuracy. The key lies in optimizing KV cache, especially in suppressing noise, providing new insights and methods for processing long contexts. Overall, this research offers valuable references for efficient memory resource utilization and model performance enhancement.