Cross‑model KV‑cache reuse is a fundamental challenge in modern LLM serving. Coding agents and multi‑model systems often need to route a shared context across different models—for example, a user may switch models mid‑session or a cascade may forward a hard query to a stronger model. Because KV caches store model‑specific representations, each switch normally forces the receiving model to prefill the entire context from scratch. Prior work shows that closed‑form linear maps can translate KV caches between models of the same family, but accuracy degrades sharply as the model‑size gap widens.
We observe that these translation failures are concentrated in a small subset of information‑dense tokens. To address this, we introduce RaReCache, a framework that lets a large target model decode accurately from a cache prefixed by a much smaller source model by selectively recomputing only the critical positions. RaReCache identifies these positions with a novel rank‑disagreement metric, which scores each token by the energy of its mapped KV in output directions that are weakly supported by calibration data.
Experiments across two model families and five benchmarks demonstrate that, for a 23× parameter gap (Qwen3‑0.6B → 14B), recomputing just 30% of the positions retains 95‑99% of the target accuracy; for an 8.8× gap (Llama3‑8B → 70B), recomputing 40% retains 96.5% accuracy. RaReCache largely eliminates sensitivity to source‑model size and achieves up to 3.04× prefill speedup. In online serving, it handles 1.8× the request throughput of target prefill on a single GPU, and at the target’s saturation load, a 30% recompute budget reduces median and 99th‑percentile time‑to‑first‑token (TTFT) by 5.0× and 6.4× respectively.
RaReCache establishes an efficient serving paradigm where small models prefill on behalf of massive targets, allowing large models to recompute only critical tokens and dramatically cut prefill latency.
Review