GPU high‑bandwidth memory (HBM) is scarce and expensive, and KV caches consume a large portion of it as chats, agent loops, and document QA accumulate state. Systems such as Mooncake, LMCache, FlexGen, InfiniGen, and AttentionStore extend GPU memory with CPU DRAM and SSD, but the hard problem is deciding which blocks belong to which tier, when to move or evict them, and whether prefetching helps. We built a discrete‑event simulator covering GPU HBM, CPU DRAM, and SSD, calibrated with a random‑forest execution‑time predictor.
Four placement policies were compared: recency, reuse frequency, predicted reuse, and an EWMA predictor with prefetch look‑ahead. The evaluation covered three workloads—chat, agent, and document question answering.
Results show tiered caching enables 73.02× more concurrent sessions per GPU and reduces cost per session by 62.04×. These gains stem from the tier capacities (1 + 8 + 64 for GPU, CPU, SSD) rather than the placement policy itself. Because decoding is compute‑bound at batch size one, placement barely affects throughput; it mainly influences PCIe migration traffic and time‑to‑first‑token.
For chat, recency cuts migration traffic by a factor of 2.30 compared with reuse frequency. Reuse frequency, however, yields the best results for agents and document QA. The predicted‑reuse policy is byte‑identical to recency, so its recommendation effectively mirrors recency. An EWMA predictor changes behavior but still ranks behind reuse frequency on all workloads.
Prefetching does not justify its bandwidth cost. Even an oracle with perfect future knowledge never beats a no‑prefetch strategy in terms of migration traffic. Workload‑specific placement can reduce data movement, but the current implementation does not support the predicted‑reuse and prefetch recommendations.
Review: The capacity distribution across tiers is the dominant factor for boosting session concurrency and cutting costs, while placement policies only fine‑tune migration traffic for specific workloads. Prefetching is not worthwhile under current bandwidth constraints; future work should focus on hardware support and more accurate predictors.