MTVA-Bench scores voice agents with transcription errors intact

MTVA-Bench grades a voice agent's language model with the speech-recognition errors still in the transcript, according to the MTVA-Bench paper.

That split matters for anyone picking a voice-agent model off a leaderboard. A score built on a clean transcript says nothing about how the model behaves once a caller's words get mangled.

Most voice benchmarks pick one failure mode. End-to-end tests blend recognition mistakes and model mistakes into one number, so a bad score doesn't reveal the cause. LLM benchmarks go the other way. They hand the model a clean transcript. That drops the conditions that make a real call hard: a caller's sentence split across two messages, a required reply language and script.

The benchmark keeps both problems in the picture. An LLM plays the caller, following rubrics and tool definitions. The agent under test still works from a transcript carrying recognition errors, split utterances, and a required reply language and script. Its score tracks which backend tool it calls and whether its multi-turn decisions hold up, not how fluent the reply sounds.

A model that reads perfectly in a clean-transcript test can still call the wrong tool once the input turns noisy. This benchmark is built to catch that gap.