Paper Finds Claude Opus 5 Often Skips Procedure To Reach Right Outcome

Researchers released BABA-Bench on 29 October 2026, a benchmark built to test whether an AI agent acting as a patient advocate not only gets the result a patient needs but gets there the right way. The team built 47 interactive worlds from de-identified real patient advocacy cases that expert advocates reviewed, each requiring an agent to find hidden information and coordinate across insurers, clinics, suppliers, pharmacies and community organisations. An engine checks every action as the agent performs it, testing whether the agent had permission, had evidence to justify the step, and whether the action was prohibited outright.

The paper evaluated 14 frontier models, running each world three times for a total of 1,974 runs. Among the 921 runs that reached every required outcome, 254 (28%) did so without following the necessary procedure somewhere along the way, such as sharing a record before securing the patient’s permission to do so.

Claude Opus 5 reached all required outcomes in 81.6% of its runs, but only 52.5% of its runs passed every procedural check. The authors also found that giving two models a verified route to follow raised pass rates sharply: one model’s pass rate rose from 29.8% to 91.5%, and another’s rose from 44.0% to 97.2%. The authors conclude that for these models, the harder problem is finding the correct route through a case, not executing a route once it is known.

The numbers

• Claude Opus 5 reached all required outcomes in 81.6% of runs but passed every check in 52.5% of runs.
• Across all 921 outcome-reaching runs, 254, or 28%, skipped a required procedural step.
• The benchmark totalled 1,974 runs across 47 worlds and 14 models.
• Giving a verified route to two models raised their pass rates from 29.8% to 91.5% and from 44.0% to 97.2%.

The gap between outcome and procedure is the finding worth sitting with. A patient advocacy agent that shares records before consent, or skips a check an auditor would expect, has not failed in a way a simple success metric would catch. For teams weighing whether to deploy an agent on tasks involving consent, evidence and prohibited actions, this paper argues for scoring the route an agent takes, not only whether it arrives.