NeFut Logo NeFut
中 Admin Login

[CS.AI] Hybrid Spark: Adaptive Speculative Decoding That Lets the Content Choose the Drafter

Published at: 2026-11-05 22:00 Last updated: 2026-11-07 12:32
#Machine Learning #optimization #LLM

Speculative decoding (SD) speeds up large language model serving by pre‑generating draft tokens on the GPU. Its benefit depends on workload shape such as batch size and context length, as well as on the generated content. A model‑based drafter emits short blocks for any content, while a model‑free drafter like an n‑gram lookup can copy long spans at almost no cost when the output repeats the input, e.g., in code editing. Adaptive verification allocates a batch‑wide verification budget to the draft tokens most likely to be accepted, but it only supports a single model‑based drafter. Model‑free drafters lack per‑token confidence, cannot participate in budget allocation, and deployments must either fix one drafter or split GPU resources between them.\n\nWe introduce \sys, which runs a model‑based and a model‑free drafter at every step and selects the better chain for each request, all on the GPU while preserving asynchronous scheduling. The key idea is a price per verified draft token: the goodput that slot would earn elsewhere in the batch. Once the model‑free drafter’s tokens receive a one‑time acceptance calibration on general text, this price puts both drafter chains on a common scale.\n\nIntegrated in vLLM with \dspark and an n‑gram drafter, experiments show that for code‑editing workloads at batch size 4 the output throughput improves by 1.10–1.62×; at batch size 128 the gain is 1.04–1.27×; and for chat at temperature 0 the performance stays within 0.97–1.01× of \dspark. A single deployment can therefore serve both workloads efficiently.\n\nReview: \sys’s unified pricing mechanism successfully merges model‑based and model‑free drafting strategies, enabling adaptive scheduling on a single GPU and markedly boosting throughput and resource utilization across diverse scenarios.

Original Source: https://arxiv.org/abs/2611.05611

[h] Back to Home