Hardware accelerators are now on the critical path of online serving, including GPUs, FPGAs, and increasingly remote services like hardware security modules, post-quantum KEMs, and inference servers. For fine-grained offloads (microseconds to a few milliseconds), traditional solutions fail: context switches are as costly as the offload, and busy-waiting burns core resources.
The fix is to overlap offloads with other requests. Previous systems achieve this by adding concurrency—through async-framework rewrites, new runtimes or data-plane OS, or hand-tuned integrations. However, concurrency already exists: serving concurrent requests involves suspending and resuming them, meaning every server has the machinery needed for overlap. Thus, overlap becomes a routing problem, not a rewriting one: submit the offload to an executor, suspend the request using the server's deferred-response primitive, and resume upon completion.
Testing across ten off-the-shelf servers covering every production concurrency model, this approach requires only 22-138 lines of code added, with at most one modification, yielding a performance gain of 1.2-5.4x on real hardware; the server's concurrency model and the offload's weight can predict both numbers in advance, bounded by device throughput and the server's overlap capacity. In the limit, an LD_PRELOAD fiber runtime can inject rerouting into an unmodified thread-per-connection binary, achieving a 17.3x performance boost within a characterized envelope. Rerouting suspends run-to-completion atomicity; a measured taxonomy confines hazards to unlocked shared aggregates, and a transparent page-protection detector safeguards these, validated on stock Redis.