Research on decoding‑time KV cache compression has largely emphasized designing better token scoring functions, while the temporal rule that aggregates scores across decode steps is often treated as an implementation detail. In aggressive KV compression scenarios we find that exponential moving average (EMA) aggregation makes scorer modifications that roughly preserve order virtually indistinguishable at the eviction‑set level. Specifically, value‑norm and entropy variants remain highly correlated with attention and produce almost unchanged retention sets, whereas KeyDiff, key norm, recency, and a learned scorer substantially alter the ranking and degrade performance. We attribute this stability to the evaluated aggregation, which couples layer weighting with temporal retention. Building on this observation we introduce InertiaKV, an EMA‑based decoding‑time eviction method, and InertiaKV‑Lazy, its periodic‑refresh variant, which achieves 1.34‑1.46× higher decode throughput compared to full‑refresh InertiaKV on the same hardware. We also study Score‑Free decoding as a separate operating point: it scores the full context once at the first decode step, freezes that ranking, and removes all subsequent scoring, yielding an average quality change of $+0.03$ while eliminating later scoring overhead. Experiments across six open‑weight backbones and the LongBench, LongBench‑v2, and RULER benchmarks identify temporal aggregation and ranking preservation as distinct, consequential design factors, without implying that scoring quality is irrelevant in general.
Review