State‑space models (SSMs) read and write a fixed‑size recurrent state at every decoding step, which becomes a bottleneck for hybrid models. The eager state I/O has low arithmetic intensity, offers no cheap rollback for speculative decoding, and propagates quantization error into later recurrent updates. ReplaySSM unrolls the recurrence into a checkpoint plus a buffer of recent SSM inputs, representing the state as a checkpoint and input buffer. Output computation is separated from checkpoint updates; the checkpoint is refreshed periodically, so most steps avoid state writes and the dominant state memory traffic is roughly halved. Keeping recent inputs turns speculative rollback into a simple buffer‑pointer move and enables output‑only verification without materializing intermediate states. The same separation lets a low‑precision checkpoint serve output computation on most steps while a higher‑precision checkpoint handles recurrent updates, preventing low‑precision rounding error from entering later updates. We implement ReplaySSM in vLLM for Mamba‑2 and Gated DeltaNet. At batch size 256 it speeds up standard decoding by $1.20$–$1.47\times$ over a tuned vLLM baseline across four hybrid models, and delivers about $1.8\times$ the throughput of vLLM’s speculative decoding on two 120B‑class models. Separate state precisions also reduce output KL divergence and downstream accuracy loss.
Review