LLM‑based multimodal emotion reasoning systems achieve strong results on standard benchmarks, yet improving them through instruction tuning often incurs high computational cost.
Agentic pipelines built from frozen models avoid this expense, but introduce a new difficulty: when a prediction is wrong, the system must pinpoint which agent or stage caused the error.
Our proposed framework consists of several key components:
- Modality‑specific perception agents process visual, acoustic, and lexical inputs, generate evidence‑cited observations, and post them to a shared board.
- An integration agent combines these observations with retrieved multimodal examples to produce an emotion description, a set of labels, and supporting citations.
- An MLLM critic can request a single revision before the output is finalized.
For incorrect predictions, an evidence‑grounded verification stage performs two checks:
- Deterministic checks ensure that cited evidence and references are valid;
- An entailment check determines whether the final claims are sufficiently supported by the evidence. Detected failures are grouped by responsible component and failure type, transformed into targeted prompt patches, and accepted only after validation on a held‑out set, enabling prompt adaptation without updating model weights.
On the MER2025 dataset, targeted prompt adaptation outperforms the frozen static pipeline across four MLLMs of varying scale (VideoLLaMA3‑2B, Phi‑3.5‑vision‑instruct, Qwen2.5‑VL‑7B‑Instruct, InternVL2.5‑38B), improving F‑score by 0.6–2.2 points. Notably, Qwen2.5‑VL‑7B rises from 52.3 to 54.5, and the approach matches or exceeds undifferentiated reflection baselines.
Review: The approach improves multimodal emotion reasoning without fine‑tuning model weights, delivering notable accuracy gains and better traceability.