Sampling in LLM inference involves a combinatorial set of logit processing, token selection, and verification operations for speculative decoding. Existing implementations accelerate only subsets of this pipeline, rely on multiple kernel launches, or assume homogeneous sampling behavior across a batch, limiting support for dynamic serving workloads and preventing efficient CUDA Graph execution.
We introduce SonicSampler, a unified suite of tile-aware Triton kernels that vertically fuses the complete sampling pipeline into a fixed, workload-aware execution model.
Our kernels support dynamic per-request sampling behaviors, including grammar-constrained decoding, repetition, frequency and presence penalties, logit bias, temperature scaling, top-$k$ / top-$p$ / min-$p$ filtering, and speculative verification - all within a single batched kernel while remaining fully CUDA Graph-compatible.
Central to our approach is a novel hierarchical two-stage top-$k$ algorithm that achieves up to 10x speedup over competitive baselines and exploits the low-entropy structure of LLM outputs for efficient selection over large vocabularies.
Across heterogeneous speculative decoding workloads, SonicSampler achieves up to 16x speedup over state-of-the-art baselines while preserving flexible batched execution.
Blogger's Review: SonicSampler represents a significant advancement in LLM sampling efficiency. By integrating multiple sampling behaviors into a single kernel, it not only optimizes performance but also enhances flexibility for dynamic workloads. The hierarchical top-$k$ algorithm is particularly noteworthy for its ability to leverage low-entropy structures, making it a game-changer for large-scale implementations.