Distributed Transformer inference on co‑located edge devices is limited by communication, because sequence‑parallel partitions must exchange activations, which dominates latency. Existing compression methods reduce bandwidth but still require the receiver to perform key/value projection, serializing computation behind communication.
Trace inserts a Vector Quantizer (VQ) after the post‑attention residual of each Transformer layer and precomputes codebook‑indexed key/value representations. During inference, activations are encoded into compact codebook indices; the receiver recovers keys and values by table lookup, avoiding downstream projection computation. This removes the downstream compute on received data, enabling overlap of communication and computation and eliminating a key latency bottleneck. A hierarchical clustering organizes the learned codebook into a multi‑resolution hierarchy, allowing runtime bitrate adaptation without retraining.
The framework supports two variants: one prioritizing accuracy, the other balancing accuracy and efficiency with buffer‑compatible quantization. Experiments on ViT‑Base (ImageNet‑1K, CIFAR‑100) and ViViT‑B (UCF‑101) show Trace achieves up to 1.89% higher accuracy and up to 1.27× end‑to‑end speedup over the state‑of‑the‑art VQ‑based distributed inference framework, while reducing communication overhead by up to 22,624× compared with uncompressed sequence‑parallel inference. Review