When an LLM serving deployment runs out of KV‑cache space, two well‑established remedies are tensor parallelism and KV compression. Tensor parallelism shards model weights and the KV cache across 2, 4, or 8 devices, freeing memory at the cost of an all‑reduce on every layer and a hardware bill that scales linearly with device count. KV compression, on the other hand, shrinks the cache in‑place on a single GPU via quantisation and eviction, sacrificing a bit of quality but keeping the hardware footprint minimal.
Prior work usually reports memory‑saving ratios for compression and throughput gains for parallel scaling, but rarely places both on the same cost axis. This paper builds a simulator calibrated on A100, A40, and H100 GPUs and maps tensor‑parallel configurations (degree 1‑8) and KV‑compressed settings (16/8/4‑bit, keep‑ratios down to 0.25) onto a cost‑normalised plane: cost per million tokens versus latency. The goal is to find a cost‑equivalence crossover point.
The study spans two models (Llama‑2 7B and 70B), three GPU types, and every feasible memory‑relief level. No crossover is found: across all scenarios KV compression is cheaper, saving between 1.20× and 2.00×. A 7B model on an 80 GB card cannot even exhaust its KV budget within its own context window. The decisive factor is model size relative to device memory, roughly 36 B parameters for an 80 GB card. Below this wall compression dominates and extra GPUs become largely wasted spend; above it tensor parallelism ceases to be optional and becomes an entry ticket. For example, Llama‑2‑70B is infeasible on a single A100 under any KV setting because the binding resource shifts to weights, which KV compression does not affect.
In terms of latency, tensor parallelism is the only lever that improves per‑token speed, while compression worsens latency by 8%–93% due to batching contention. Regarding capacity‑per‑dollar, compression multiplies it by about 16.5×, whereas an eight‑fold GPU spend yields only ~1.21×.
Blogger's Review: By unifying the cost perspective, the paper convincingly shows that, for memory‑bound LLM serving, KV compression usually offers far better value than simply adding more GPUs, and tensor parallelism only becomes necessary when the model size outgrows the device memory.