As sequence lengths keep growing, distributed attention based on sequence parallelism (e.g., Ring‑Attention) has become the mainstream approach for large language models. However, inherent communication dependencies prevent overlap between computation and communication, limiting GPU utilization. In cross‑hardware settings such as NUMA or multi‑node clusters, bandwidth imbalance causes frequent synchronization and hampers overall performance. To address these issues we introduce CHARM+, a cross‑hardware attention method with a re‑merge multistream mechanism. The approach reorganizes sequential communications into multiple levels, enabling intra‑level parallelism that overlaps with computation, and decouples cross‑hardware communication from intra‑hardware communication to remove synchronization overhead caused by bandwidth disparity. Forward and backward kernels were implemented on various hardware platforms and consistently outperformed existing techniques. Compared with Ring‑Attention, CHARM+ achieves roughly 1.3×/1.7× speedup for forward/backward on a dual‑NUMA PCIe system, and about 2.8×/2.9× on a dual‑node NVLink system.
Review: By restructuring communication into hierarchical streams, this work markedly improves the efficiency of cross‑hardware attention, offering a scalable path for training ever larger models.