NeFut Logo NeFut
Admin Login

[CS.AI] MTVA-Bench: Evaluating the Language Model Inside Cascaded Voice Agents

Published at: 2026-09-18 22:00 Last updated: 2026-09-20 12:54
#Machine Learning #LLM #Artificial Intelligence

Most voice assistants are built as cascaded pipelines: an ASR model transcribes the caller’s audio, a language model reads the transcript to decide what to say and which backend tools to invoke, and a TTS model vocalizes the reply. All decision‑making resides in the language model, yet current evaluations are either too coarse—scoring the entire pipeline and mixing transcription errors with model errors—or too narrow—isolating the model but ignoring real‑world call challenges such as transcription glitches, speaker splits across turns, and strict script compliance.\ \ We introduce the Multi‑Turn Voice Agent Benchmark (MTVA‑Bench), which evaluates the language model under the exact conditions it faces inside a cascaded system. The caller is simulated by an LLM that follows a set of rubrics, and tool calls are answered by a mock backend that responds to the actual arguments the model sends. The benchmark comprises 49 agents, 490 vetted scenarios, and supports seven languages.\ \ Scoring combines deterministic checks on tool calls with two LLM judges. One judge scores scenario‑specific rules, the other grades conversation quality without seeing the task description. Both judges must cite specific messages from the transcript. Task and conversation scores are weighted equally, because a call can complete its task yet still be a poor experience for the user.\ \ In a study of seven models, six models differ by only 6.4 points on correct tool selection, but their overall scores span 24.4 points. The gap stems mainly from argument values, action ordering, rule compliance, and what the model says around its tool calls.\ \ Review

Original Source: https://arxiv.org/abs/2609.20152

[h] Back to Home