We present a full-pipeline inference optimization for the MiMo-V2.5 model family, integrating Hybrid Sliding Window Attention (Hybrid SWA), sparse Mixture-of-Experts (MoE), and multimodal encoders. While Hybrid SWA ideally reduces both attention computation and KVCache storage compared to Full Attention, achieving these benefits in production necessitates significant engineering efforts.
We systematically optimize the KVCache system with layerwise prefetch, SWA-aware prefix cache trees, and specialized placement strategies, achieving strict $O(W)$ SWA storage and high cache hit rates. Additionally, we build GCache, a high-performance distributed cache infrastructure with RDMA-optimized networking, and develop a KVCache-affinity router to minimize computation while maintaining load balancing. We also optimize for multimodal inputs, including GPU image preprocessing, parallel video decoding, and multimodal cache sharing. Together, these optimizations constitute the first large-scale LLM serving system in production that efficiently covers the Hybrid SWA + MoE + multimodal composite architecture.
Blogger's Review: This study significantly enhances the inference efficiency of the MiMo-V2.5 model through systematic KVCache optimizations and the introduction of a high-performance distributed cache infrastructure, demonstrating the engineering potential in complex architectures and providing valuable insights for practical LLM applications.