In the inference process of Large Language Models (LLMs), prefix caching can reduce inference latency by reusing KV caches across requests with shared prompts. However, cluster-scale reuse faces challenges due to the partitioning of caches across nodes. This paper proposes a decentralized, prefix-cache-aware routing scheme for peer-to-peer LLM serving.
Each node maintains a local radix tree of its cached prefixes and asynchronously refreshes estimates of peer caches using periodic anti-entropy. Requests are routed to the node with the longest estimated prefix match, without centralized coordination or KV-cache transfer. Stale metadata only causes cache misses, not incorrect outputs, making weak consistency sufficient for correctness.
Evaluation on simulated MMLU workloads shows that decentralized routing improves latency under low communication delay and skewed prefix distributions, while high network latency and affinity-induced hotspots limit its benefits.
Blogger's Review: This study highlights the potential of decentralized routing in LLM inference, particularly in optimizing cache utilization. However, the impact of network latency and node distribution on practical applications still needs further exploration to ensure system scalability and efficiency.