In the evaluation of KV-cache compression methods, queries are typically appended to the context before compression, following a query-aware protocol. However, the economic rationale for a compressed KV cache lies in reuse: compress a document once to answer many future questions. Therefore, compression must occur in a query-agnostic manner, prior to any question being seen.
We conducted a matched-budget audit of six published compression methods against three trivial baselines on three open 7-9B models (144,300 paired evaluations on RULER-8192; 40,800 on LongBench; and 50,000-resample paired bootstrap throughout). Everything was kept fixed—model, compression ratio, instances, decoding—except for the scoring rule.
Three key findings emerged:
-
Query visibility alters the rankings: under the agnostic protocol, of the five audited methods sharing a common attention backend, only KeyDiff consistently outperformed a best-of-3 trivial baseline in 31 of 36 cells, whereas the widely deployed SnapKV method averaged a loss to the “keep the start and the recent window” method (-0.066).
-
The drop in performance for each method between the two protocols is consistently ordered by how visible the question is to each method's scoring signal, as reflected in their source code: from Delta=+0.198 for SnapKV (where the question resides within its 64-token observation window) to Delta=+0.011 for KeyDiff (which contains no query term at all).
These findings highlight the significance of query visibility in the evaluation of KV-cache compression and its impact on the performance of different methods.
Blogger's Review: This article provides a deep dive into how query visibility affects the rankings of KV cache compression methods through a matched-budget audit. The insights reveal the importance of compression strategies in practical reuse scenarios, offering valuable references for further optimizing compression algorithms.