Google's Gemini Live Benchmarked in New Voice-Agent Stress Test

A benchmark published on arXiv on September 29, 2026 gives the clearest public reading yet of how voice agents behave once a call stops being a clean, quiet demo. VAmoS Energy, built by a team including Joshua Meyer and Sahar Shayegan, runs 100 simulated calls about utility billing and payment assistance, each with a caller who makes two to four separate requests in one conversation. The agent under test carries sixteen tools, wired to a stateful Stripe billing twin and the Apache Fineract loan engine, and account access stays locked until the caller passes verification. Task data comes from public household electricity records, and the billing policy is modeled on Pennsylvania’s residential rules.

What sets this benchmark apart from an ordinary transcript check is that it grades two different things at once: what the agent says out loud, and what it actually changes in the underlying account. An LLM-as-a-verifier compares both against explicit written requirements, and on a calibration run it agreed with a separate code-based verifier on 99.1 percent of checks, giving the authors a reason to trust the automated grading at scale.

Fourteen voice stacks were run through the full test, three repeats per task, and completion rates varied widely, from 17.3 percent to 44.7 percent across the pool. Grok Voice came out ahead overall, while Google’s Gemini 3.8 Live and GPT-Live followed close behind each other at roughly similar cost per call. The paper’s most striking single result is what happens when a television plays in the background during the call: pooled completion fell from 38.7 percent down to 8.6 percent, a collapse the authors attribute partly to callers who accept an incorrect result simply because they heard words that sounded right, with no way to inspect what the agent had actually done to their account.

The numbers

• 99.1% agreement between the automated verifier and a code-based check, the figure used to justify scaling the grading.
• 17.3% to 44.7% was the completion range across the fourteen voice stacks tested, three runs per task.
• 8.6% was pooled completion once background television noise was added to calls, down from a much higher baseline without it.

For teams evaluating Gemini 3.8 Live or any rival voice stack for a billing or payments workflow, this benchmark argues for testing under noise and with multi-step requests before signing a contract, not after a smooth-sounding pilot call.