Abstract
Weak supervision sets a practical regime for audio-visual sound source localization as dense spatial annotations are costly to obtain at scale. The task, however, remains challenging, as models must locate sound sources from temporally aligned audio-visual data without pixel-level supervision. Recent large-scale audio-visual retrieval models, trained at unprecedented scale, encode rich multimodal structure. We show their latent representations, though optimized for global alignment, can nonetheless enable fine-grained spatial grounding. While spatial detail is progressively lost in the upper layers of retrieval backbones due to global pooling, intermediate visual tokens retain highly structured spatial information.
To exploit this, we introduce LAIP (Localization via Audio-Informed Pooling), a framework that employs a lightweight Audio-informed Spatial Pooling (AiSP) to replace the standard global aggregation module. By using frame-aligned audio to query intermediate visual tokens, LAIP recovers localized spatial information that is otherwise discarded by the frozen retrieval pipeline. Our approach achieves state-of-the-art performance on AVSBench and AVATAR, nearly doubling previous results on the latter. These findings prove that accurate localization does not need to be learned from scratch; instead, it can be unlocked from existing retrieval representations, providing a unified path for both retrieval and localization tasks.
Blogger's Review: This research demonstrates how to leverage the latent representations of existing audio-visual retrieval models to enhance spatial grounding capabilities, significantly advancing the field of audio-visual processing. The LAIP framework, through its intelligent audio-informed strategy, breaks the limitations of traditional global pooling and shows great practical value. In the future, combining more modalities could further enhance the model's performance.