Ambivalence and hesitancy are subtle behavioral states expressed through a combination of verbal content, facial behavior, visual context, and acoustic cues. Effective recognition thus requires not only extracting informative unimodal representations but also modeling how temporally aligned behavioral evidence interacts across modalities. This paper proposes a synchronized visual-facial cross-refinement framework (SVF-CR) with pairwise multimodal evidence fusion for ambivalence and hesitancy recognition.
The proposed method first extracts whole-video segment tokens and cropped-face segment tokens using the same temporal partition. The synchronized visual and facial tokens are refined through intra-modal self-attention and bidirectional visual-facial cross-attention, allowing whole-video context and local facial behavior to mutually refine each other before evidence construction.
We then construct segment-level visual-facial evidence using consistency and discrepancy modeling, followed by temporal self-attention and attention pooling. Textual and acoustic features are lightly refined through context self-attention and are fused with the enhanced visual-facial evidence at the final decision stage using pairwise evidence fusion. Experiments on the BAH (Behavioral Ambivalence/Hesitancy) public evaluation split show that the proposed synchronized visual-facial cross-refinement improves public macro-F1 over both global visual-face token fusion and synchronized evidence baselines, achieving a public macro-F1 of 0.7156.
Code is available at: GitHub - SVF-CR.
Blogger's Review: This study effectively integrates multimodal evidence through the innovative SVF-CR framework, providing new insights for recognizing ambivalence and hesitancy. The method not only enhances recognition accuracy but also lays a foundation for future research in multimodal emotion analysis, demonstrating significant potential applications.