Tensor parallelism ends each layer with an all‑reduce placed between two dependent matrix multiplications, so its cost cannot be hidden and grows as compute outpaces interconnect bandwidth. Sending values in 4 bits instead of 16 bits can cut this cost by nearly fourfold, but the all‑reduce must add the carried values, and quantizing the sum injects an error at every call. Model quality depends on how these errors accumulate across layers, yet existing methods treat each call in isolation.
Echo introduces a 4‑bit all‑reduce for both training and inference that controls the accumulated error. Data are moved in 4 bits while summation is performed in high precision, ensuring each value is quantized at most twice per call regardless of the number of GPUs. The summation reveals each injected error, and Echo feeds it back into later calls so that old errors cancel instead of accumulating. We prove that the accumulated error remains bounded independently of depth.
On A100 GPUs, Echo is 2.09× faster than the 16‑bit NCCL all‑reduce for 1 GB messages. It speeds up time‑to‑first‑token by up to 1.16× and training step time by up to 1.09× on an idle interconnect, and by up to 1.44× and 1.38× as background traffic grows. The average accuracy over seven benchmarks stays within 0.4 points of 16‑bit inference. In a 10 B‑token pre‑training run, its final loss is less than 0.9 % above that of 16‑bit training. Echo requires no FP8 or FP4 hardware.
Review: Echo achieves low‑bit communication without sacrificing model quality by compensating quantization errors, offering a practical speedup for large‑scale LLM training and inference.