Paper Benchmarks Grok's Voice Agent Across Five Languages

Researchers extended an existing English voice-agent benchmark, tau-Voice, into a new multilingual version called tau-Multilingual, adding Spanish, Brazilian Portuguese, Hindi, Korean and Mandarin with native speaker review of both the generated language and the spoken output. The paper was submitted on September 21, 2026, and evaluates several voice agents, including Grok, on 4,500 full-duplex phone calls spread across five voice configurations.

The results split cleanly into two groups. Spanish, Portuguese and Hindi stayed within 3.2 task-completion points of English performance, a gap the authors treat as close enough that those languages behave much like the English baseline. Korean and Mandarin did not: Korean task completion fell by 14.7 points and Mandarin by 8.4 points against the same English reference. The paper also separates out why each language struggled rather than reporting one number. Korean systems missed more responses outright, while Mandarin systems interrupted callers more often mid turn, and both languages produced more errors handling tools and named entities than the higher-scoring group did.

Grok’s own results cut against a simple headline. The paper reports that Grok leads on task completion across the tested systems, meaning it gets the call’s job done more reliably than the other agents evaluated. But on generation quality, the score covering how natural and accurate the spoken language itself is, Grok scores lowest among the systems tested. The authors argue this split is exactly why task, interaction and generation need to be reported as separate numbers rather than folded into one score: a system can finish a caller’s request while producing speech that a native reviewer would mark down.

Alongside the benchmark, the authors say they are releasing the language packs, the validated judges used to score each call, and supporting tools so other teams can build and run their own multilingual voice-agent evaluations rather than depending on a single lab’s English-only numbers.

The numbers

The 4,500 calls, the 3.2-point near-English cluster, and the 14.7 and 8.4-point drops for Korean and Mandarin are the core comparison points a reader evaluating a voice agent for non-English markets needs before trusting any single blended accuracy claim a vendor publishes.