Multimodal large language models (MLLMs) excel at visual question answering and scene understanding, yet they remain vulnerable to shortcut behavior in affective reasoning. Models can produce correct answers while ignoring people‑centric cues such as micro‑expressions and body language, which harms traceability and external verification. Existing reinforcement‑learning approaches mainly reward contextual or logical coherence without explicitly enforcing attention to human evidence; moreover, using an LLM as a judge often leads to score clustering, reducing reward discriminability.
AffectOmni introduces a GRPO‑based training framework for verifiable affective reasoning. It adds People Focus and Temporal Order rewards to encourage selection of human‑centric evidence and temporally structured reasoning. The reward is formalized as $R = R_{PF} + R_{TO}$, where $R_{PF}$ measures alignment with human evidence and $R_{TO}$ evaluates temporal consistency of reasoning steps. To obtain stable and discriminative signals, a within‑group comparative scoring scheme is employed, comparing samples inside the same batch rather than using absolute scores.
For verification, a Thinking Summarizer converts free‑form rationales into executable evidence instructions, which are grounded to pixel‑level evidence regions via SAM3, providing an auditable interface outside the training loop.
Experiments on IntentBench, Daily Omni, and WorldSense show consistent gains over open‑source 7B‑scale baselines: +4.66% on emotion recognition and +14.29% on temporally sensitive tasks. The code is released at https://github.com/eliot127825-rgb/AffectOmni_nobody.
Blogger's Review: AffectOmni’s reward design and auditable evidence grounding address the interpretability gap in affective reasoning, offering a promising direction for the safe deployment of multimodal models.