NeFut Logo NeFut
Admin Login

[CS.AI] Dual-Flow Transformers: Decoupling Primary Prefill Path from Additional Decode Computation

Published at: 2026-08-14 22:00 Last updated: 2026-08-16 07:03
#AI #Machine Learning #Transformer

As large language models serve more requests, cumulative inference cost is becoming increasingly important relative to one-time training cost. The two inference phases stress hardware differently: prompt prefill is parallel and typically compute-bound, whereas autoregressive decode is sequential and often memory-bandwidth-bound. We introduce the Dual-Flow Transformer, which decouples the primary prefill path from additional decode computation. The primary flow is a complete causal language model that processes the prompt and writes the key-value (KV) cache. The auxiliary flow is omitted during prompt processing and activated only from the final prompt position onward, adding continuation-prediction computation without writing persistent state or influencing the primary flow. The two flows share major attention, MLP, and output matrices, while using separate token embeddings and lightweight coupling. Across matched-token comparisons, Dual-Flow achieves lower validation loss across architectures and data configurations. Blogger's Review: The Dual-Flow Transformer decouples the primary prefill path from additional decode computation, achieving better inference efficiency and lower validation loss. This technique has the potential to be widely applied in large language models, improving model performance and efficiency.

Original Source: https://arxiv.org/abs/2608.12385

[h] Back to Home