Abstract
The inference efficiency of diffusion large language models (dLLMs) is constrained by two challenges: bidirectional attention precludes efficient KV-cache reuse, while increasing decoding parallelism with static confidence thresholds can compromise generation quality.
We observe that both challenges arise from a shared phenomenon: as tokens are decoded, their contextual integration through bidirectional attention causes token representations to drift (evolve) across decoding steps. This insight motivates Polestar, a training-free inference framework that uses token representation drift as a unified signal to jointly address both challenges.
Polestar comprises two components:
- Polestar-Cache: identifies stale KV-cache positions via drift and performs sparse KV-cache refreshes to enable efficient reuse.
- Polestar-Commit: detects sharp drift events to reliably identify commit-ready tokens.
Across mathematics and coding benchmarks on several dLLM families, Polestar sets a new state of the art on the accuracy-throughput Pareto frontier, achieving up to 10.73% accuracy improvement, up to 3.7x higher throughput, and high decoding parallelism of 3.67 tokens per forward pass over existing baselines.
Blogger's Review: Polestar cleverly addresses the efficiency bottlenecks in traditional inference by focusing on token representation drift, showcasing its potential in practical applications. This innovative approach not only enhances inference performance but also provides a new research direction for future LLM optimizations.