Recent breakthroughs in Reinforcement Learning (RL) have highlighted its potential for complex agentic Large Language Model (LLM) tasks. However, existing efforts primarily focus on single-task settings, while real-world applications require a generalist agent capable of solving multiple tasks simultaneously. In this work, we identify a critical yet underexplored phenomenon in multi-task agentic RL: exploration-exploitation pace mismatch across different tasks. Specifically, easier tasks may converge early to low-entropy policies that hinder learning on harder tasks, while harder tasks can push easier tasks back toward high-entropy exploration. This interaction creates inter-task entropy crossovers and frequent entropy spikes. Inspired by this observation, we introduce Entropy Pacing Policy Optimization (EPPO) for multi-task agentic LLMs, which coordinates entropy across tasks to stabilize multi-task optimization. At the core of EPPO is a task-wise dynamic clipping mechanism that replaces the fixed clipping threshold in Group Relative Policy Optimization (GRPO) with a task entropy-aware adaptive bound, tightening updates for over-confident tasks while relaxing them for under-explored ones. Experiments on multi-task agentic benchmarks demonstrate that the proposed EPPO yields superior results compared to its counterparts.
Blogger's Review: This paper delves into the critical issue of entropy management in multi-task reinforcement learning. By introducing the EPPO mechanism, it effectively addresses the pace mismatch between tasks, providing a fresh perspective for multi-task learning. This not only enhances the model's learning efficiency but also points the way forward for future research.