Autoregressive Transformer decoding on tiled accelerators is limited by irregular KV cache movement. Prior compression and DRAM‑placement schemes still funnel traffic through centralized memory paths, creating a bottleneck for long‑context serving. MeshKV packets KV blocks and routes them over a lightweight NoC, while three co‑designed mechanisms improve efficiency. TaKV uses affine striping to scatter KV blocks across nodes, reducing hotspot load. Mare multicast adds verified duplicate suppression to avoid redundant transfers. Pad overlaps prefetch, tile multiplication, and streaming softmax behind credit‑aligned FIFOs, turning bisection back‑pressure into useful KV transfer. On an 8×8 FPGA with LLaMA‑2‑7B and Mistral‑7B at 8K‑32K context, MeshKV cuts interconnect traffic by up to 58%, boosts KV bandwidth utilization by 2.1×, and raises multi‑stream throughput to 1.9×.
Review: MeshKV substantially mitigates KV access bottlenecks while preserving compute density, offering a scalable hardware pathway for large‑model decoding.