Emotion Recognition in Conversations (ERC) aims to identify speakers' emotions across multi‑turn dialogues. Accurate emotion detection can empower empathetic chat agents, mental‑health support, and educational technologies. Most recent approaches rely on task‑specific fine‑tuning, which may exploit dataset‑specific cues. A largely unexamined assumption in ERC is that each utterance carries a single, unambiguous emotion label.
To test this assumption, we evaluate Large Language Models (LLMs) in a zero‑shot setting, feeding them preceding conversational turns as context. Our findings show that aggregate metrics hide systematic failures: errors cluster around utterances containing negations, exclamations, and interjections. This pattern persists across all evaluated models, suggesting that the issue lies more with the benchmarks than with any particular model.
A controlled re‑annotation study with four human annotators supports this view: strong agreement occurs in only 35% of cases, with neutral utterances dominating high‑agreement instances, while many emotional categories fall into low‑agreement regimes. These results indicate that many apparent model errors actually stem from annotation ambiguity rather than poor emotion understanding.
Consequently, standard single‑label evaluation is insufficient. To address this limitation, we introduce an LLM‑as‑Judge framework that evaluates each emotion independently for its plausibility within the conversational context, rather than enforcing a single‑label decision.
Review