MTVA-Bench evaluates the language model inside a cascaded voice agent, with ASR noise left in

arXiv·medium signalXBlueskyLinkedInCopy link

arXiv 2609.20152 (submitted 2026-09-17) targets a gap in voice-agent evaluation: end-to-end benchmarks mix recognition errors and model errors into one score, while LLM benchmarks isolate the model but drop the conditions that make phone calls hard. MTVA-Bench runs the language model under the conditions it actually faces inside a cascade, with transcription errors, a caller's utterance split across messages, and a required reply language and script. The caller is played by an LLM following rubrics and tool definitions, so the same model can be scored on multi-turn decision making and backend tool calls rather than transcript fluency.