NeFut Logo NeFut
中 Admin Login

[CS.AI] Binaural Sounding Instance Disambiguation for Audio-Visual Segmentation

Published at: 2026-11-05 22:00 Last updated: 2026-11-07 12:32
#AI #Machine Learning #Neural

Existing audio‑visual segmentation methods usually predict sound‑related regions or category masks, and they often fail to pinpoint the true sounding instance when multiple same‑class candidates appear. Recent instance‑level approaches rely heavily on motion cues and cross‑frame association, which become unreliable under sparse observations or subtle motion. To address this, we incorporate binaural audio, whose interaural time and level differences encode source direction, offering spatial constraints for distinguishing the sounding instance among same‑class visual candidates. We propose the Binaural Sounding Instance Disambiguation (BiSID‑AVS) framework, which progressively refines category‑level candidate responses into a mask of the true sounding instance. BiSID‑AVS first grounds category‑consistent candidate regions via audio prompting, then projects binaural cues onto an image‑plane spatial response map, and finally refines the true instance through dual‑level spatial conditioning at both feature and decoder stages. Experiments on the binaurally extended AVS‑Synthetic benchmark and the Habitat‑AVS dataset for robotic indoor perception show that BiSID‑AVS improves instance‑level mIoU by 35.19% and 41.21% on multi‑candidate subsets, while substantially reducing false activations on non‑sounding same‑class regions.\n\nReview: The method leverages binaural directional cues to overcome the limitations of motion‑based association, achieving precise disambiguation of same‑class candidates and advancing instance‑level audio‑visual segmentation.

Original Source: https://arxiv.org/abs/2611.06074

[h] Back to Home