Model serving accounts for a major portion of cost in production recommender systems. Maximizing throughput requires fine‑grained tuning across a deep hierarchy—GPU kernels, the ML‑framework computation graph, the model server, and on‑demand feature processing—each demanding specialized expertise. Acquiring cross‑layer knowledge is difficult and does not scale with growing workloads, leaving substantial efficiency gains untapped.
Recent AI agents have reached near‑human expert performance on standalone GPU kernel optimization, yet automated tuning for the rest of the serving stack remains largely unexplored. FlashVector addresses this gap by providing an agentic system that optimizes performance across every layer of the model‑serving stack. The key contribution is an extensible framework that generalizes the single‑kernel optimization paradigm to heterogeneous technical stacks, enabling holistic improvements.
After deployment on Unity’s Vector advertising platform, FlashVector achieved up to 2× throughput increase and 1.98× latency reduction on the model server, and up to 1.6× throughput gain on the feature store. These gains were discovered not only at the GPU kernel and computation‑graph levels but also within the model server (NVIDIA Triton C++ codebase) and the on‑demand feature transformation service (Python codebase), demonstrating the framework’s extensibility to more complex system architectures.
Review: FlashVector proves that cross‑layer automated optimization is feasible and offers a promising avenue for cost‑effective scaling of large‑scale recommendation services.