Abstract
Inspired by the design of client caching in Content Delivery Networks (CDNs), PTStore distributes and replicates popular tensors that form reusable KV cache prefixes, which are the main technique used by state-of-the-art approaches to accelerate inferences. This reduces the latency of accessing the KV cache and alleviates load imbalance caused by a disproportionately large number of requests on servers containing popular tensors. Furthermore, thanks to decentralization, PTStore allows the expansion of the size of the KV cache for LLM inference by orders of magnitude. As a result, PTStore can execute inferences on long passage Q&A datasets 5-6 times more efficiently than current baselines, which do not aggregate memory across different nodes and GPUs and therefore require regenerating the KV cache.
Blogger's Review: The design of PTStore cleverly draws on the successful experiences of CDNs, effectively enhancing the performance of inference services through a distributed architecture. This approach not only optimizes the caching mechanism but also addresses server load imbalance, providing robust support for large-scale inference tasks and showcasing the potential of decentralized technologies. This technology undoubtedly paves the way for future model inference advancements.