Automatically recognizing sentiment, whether positive or negative, from speech is a challenging task that requires the analysis of vocal inflections and the interpretation of uttered words. Recent solutions rely on audio foundation models, but it remains unclear if these models can account for all aspects. To address this, we propose a cross-modal solution that integrates audio and text information via cross-modal transformers, where text transcripts are automatically generated using an automatic speech recognition (ASR) tool.
Additionally, we create multiple text modalities by translating the transcripts into various languages using machine translation tools. Audio and multilingual text features are combined through a cascaded architecture comprising cross-modal transformer blocks that integrate modalities sequentially. We further distill knowledge from the multimodal model, termed teacher, into a unimodal (audio-only) model, referred to as student.
Experiments on a large-scale dataset demonstrate that the automatically generated textual information can significantly enhance performance in multimodal sentiment polarity classification. Our ablation study confirms the utility of both automatic transcripts and automatic translations. Furthermore, we show that the audio-only model can be improved through distillation, enhancing performance without any computational overhead during inference. To reproduce the reported results, we publicly release our code at GitHub.
Blogger's Review: The cross-modal integration method proposed in this paper offers a fresh perspective for audio sentiment analysis, significantly improving classification performance through automatic translation and distillation techniques. This has high application potential, especially in multilingual contexts. The open-source code also facilitates further research in this area.