Abstract
Audio-language embedding models such as CLAP are widely evaluated on matching present sound events, but rarely on negation. Our study reveals that this affirmation-only evaluation hides a key limitation: these models fail to effectively encode negated sound concepts, mapping affirmative and negated captions to nearly identical representations.
Introducing NegEval-Audio
To expose this blind spot, we introduce the NegEval-Audio framework, which converts existing datasets into two negation-aware tasks: Retrieval-Neg and Multiple-Choice Negation (MCQ-Neg), to probe whether models can distinguish present from absent events.
Performance Analysis
On the AudioCaps and Clotho datasets, performance degrades sharply under negation, with MCQ accuracy falling far below chance, and this failure persists even for a recent multimodal LLM-based embedding model.
While a training-free steering method improves MCQ-Neg performance, it yields marginal gains for Retrieval-Neg. This indicates that affirmation bias is a fundamental flaw in the representation geometry, necessitating explicit negation-aware training objectives.
Blogger's Review: This article delves into the limitations of audio-language embedding models in handling negation, revealing blind spots in traditional evaluation methods. By introducing the NegEval-Audio framework, the authors provide a new perspective and methodology for improvement in this field, emphasizing the importance of incorporating negation awareness into model training.