Recent multimodal Speech Emotion Recognition (SER) systems achieve high accuracy by using bidirectional cross‑modal Transformers, but their computational cost hampers deployment in latency‑sensitive and resource‑constrained surveillance settings. To tackle this issue, we introduce RAFM_SER++, which centers on an asymmetric Residual Attention Fusion Mechanism (RAFM). RAFM employs a one‑directional residual attention pathway to inject affective speech cues into semantic text representations, avoiding expensive bidirectional interactions. The framework also incorporates a BYOL‑inspired cross‑modal alignment objective and attention‑guided pooling to enhance multimodal representation learning while keeping overhead low. Experiments on the IEMOCAP and ESD benchmarks show that RAFM_SER++ consistently outperforms the HuBERT‑Base baseline and offers a superior accuracy‑efficiency trade‑off compared with the state‑of‑the‑art MemoCMT. Specifically, trainable parameters are reduced by more than 60%, inference speed reaches 79.60 it/s, and BACC scores are 81.10% on IEMOCAP and 95.39% on ESD. These findings demonstrate that lightweight asymmetric multimodal fusion is an effective alternative to interaction‑heavy Transformers for real‑time surveillance applications.
Review: RAFM_SER++ achieves notable parameter savings and speed gains without sacrificing performance, making it a practical solution for emotion recognition in constrained monitoring environments.