Abstract
The performance of large language models (LLMs) increasingly relies not only on the base model but also on the inference-time controller used to organize reasoning. Existing post-training methods typically optimize for a single fixed interaction pattern, despite real deployments relying on diverse controllers such as Chain-of-Thought, self-consistency, debate, planning, and verification pipelines. This creates a training-deployment mismatch and limits transfer to new workflows. We introduce CALM (Controller-Aware Language Models), a post-training framework that explicitly incorporates controllers into the training loop.
We formulate controller-aware post-training as multi-task reinforcement learning over controller-induced interaction protocols, where controllers are compositions of reusable local reasoning modules. This structure also induces a module-level decomposition of mixed-controller training under a turn-level GRPO objective, enabling a systematic study of controller and module-aware training strategies. We evaluate CALM on held-out controller compositions and broader controller shifts, showing that controller-aware post-training improves generalization across inference-time workflows beyond single-controller optimization.
Blogger's Review: The introduction of the CALM framework presents a novel approach for the flexible application of large language models, highlighting the significance of controllers during the inference process. By employing multi-task reinforcement learning, the model adapts to diverse interaction needs, demonstrating improved generalization capabilities, thus advancing the field of natural language processing.