Multi‑token prediction (MTP) speeds up language‑model decoding by drafting several tokens in one forward pass and verifying them together in a single target‑model pass. The optimal speculation horizon varies with context: longer drafts amortize more work when they are reliable, but they waste compute if the suffix is later rejected. Unlike autoregressive drafting, where a stopping rule can be applied token‑by‑token, MTP must fix the horizon before drafting because all heads fire in the same forward pass.
We therefore formulate horizon selection as an online control problem, using verification feedback to learn how far a draft can be trusted in each context, and introduce Adaptive Multi‑Token Prediction (AMTP). AMTP is a lightweight controller that requires no retraining and leaves the verifier unchanged.
Before each drafting round, AMTP predicts the probability that a draft prefix will be fully accepted (acceptance survival), computes the expected tokens‑per‑second for each possible horizon, and picks the horizon with the highest expectation. The predictor updates after every verification pass, learning freely from the data.
Theoretically, we show that any online learner’s prediction error translates into throughput regret against the best horizon for each context, which in turn bounds wall‑clock completion time. With online gradient descent as the learner, the regret grows sublinearly.
Empirically we evaluate five draft‑target pairings built on Llama‑3.1‑8B‑Instruct with Medusa heads, Meta’s MTP models, and Qwen3‑4B‑Inst‑2507‑MTP, using the AIME 2025 and GSM8K benchmarks. AMTP achieves up to 4.1× speed‑up over standard decoding and matches or outperforms a fixed window in 8 of 10 settings, delivering throughput gains of up to 10%.
Review: AMTP’s online horizon adaptation balances efficiency and reliability without modifying the underlying model, demonstrating the promise of online control techniques for large‑model inference optimization.