Abstract
Multimodal large language models (MLLMs) have achieved impressive performance in multimodal emotion recognition (MER) tasks, elevating emotion understanding to a new level with advanced video understanding abilities and natural language description. However, existing MLLM-based methods often utilize a fixed prompt to perceive emotions, neglecting the dynamicity and complexity of the emotion sources in multimodal inputs. To address these issues, we propose a novel Reinforcement Learning-based Dynamic Agent Specialization framework (EmoAgent-R1) to optimize the emotion recognition, reasoning, and generalization abilities of an MLLM with dynamic agent specialization.
Specifically, we first adopt a cold start strategy to endow an MLLM with preliminary emotion recognition, reasoning, and agent routing ability by training with synthetic answer-conditioned chain-of-thought data and agent routing data. Then, we further train the MLLM with reinforcement learning to perceive emotions in a two-step agentic workflow involving agent selection and agent specialization. To effectively train EmoAgent-R1, we propose a novel Progressive Group-Relative Policy Optimization (P-GRPO) to combine group-based relative advantages with a PMI-inspired progressive token-level modulation, transforming sparse rewards into fine-grained learning signals and mitigating the coarse-grained uniform credit assignment issue in GRPO. Extensive experiments on MER benchmarks demonstrate the superiority of EmoAgent-R1 in stronger emotion reasoning performance and improved optimization stability.
Blogger's Review: EmoAgent-R1 significantly enhances the capability of multimodal emotion understanding through the integration of dynamic agent specialization and reinforcement learning, excelling in complex emotion reasoning tasks. Its innovation lies in effectively addressing the limitations of traditional models in perceiving dynamic emotion sources, promising breakthroughs in broader application scenarios.