Large language models (LLMs) have achieved remarkable capabilities, yet deploying them faces serious efficiency hurdles. Autoregressive decoding incurs high latency and under‑utilizes accelerators at low batch sizes; discrete diffusion models can generate in parallel but need many denoising steps to approach autoregressive quality; long‑context reasoning creates memory bottlenecks that strain even state‑of‑the‑art hardware.
This work introduces the concept of self‑orchestrating language models: during generation the model annotates semantic dependence for each token, i.e., which previous tokens it depends on. A runtime consumes these annotations to decide execution strategies—parallelizing decoding, evicting unnecessary cache entries, or ordering diffusion denoising—thereby achieving Pareto‑optimal quality‑efficiency trade‑offs while preserving output quality.
Based on this idea, three systems are built:
-
PASTA: the model learns to indicate which output chunks can be generated independently. At inference time the runtime schedules those chunks in parallel, dramatically reducing decoding latency.
-
TIP: dependence annotations reveal redundant intermediate reasoning steps in the KV cache. The runtime evicts these irrelevant contexts, cutting memory consumption with negligible accuracy loss.
-
Planned Diffusion: before discrete diffusion, the model autoregressively produces a “plan” that specifies the denoising order of chunks. The runtime then denoises chunks in parallel according to the plan, lowering the number of diffusion steps and improving the quality‑efficiency ratio.
Experiments show that all three self‑orchestrating systems achieve better quality‑efficiency balances than conventional autoregressive or diffusion approaches across various tasks, confirming semantic dependence as an effective signal for inference scheduling.
Review