StoreBench Paper Tests Seven Frontier Models Against a Scripted Store Manager

A team of three researchers released StoreBench, a benchmark that puts a language model agent in charge of a mid-size online apparel store for stretches as long as a simulated year. The paper, posted to arXiv on October 7, 2026, argues that most agentic benchmarks test a static world: the clock only moves when the agent acts, and the pass bar is picked without reference to how well the task can actually be done. StoreBench instead runs on a production-grade commerce backend where customers order around the clock, suppliers reprice or fail on their own schedule, and market shocks arrive with little or no warning, so the agent has to plan through conditions it cannot fully predict.

The agent works through 29 merchant tools, the same set a human operator would use, under a budget where simulated time advances according to actions taken rather than the model’s own response speed. That design choice means a slower model is not penalized just for thinking longer. Seven frontier models were tested across 11 scenarios running from 30 to 45 days, plus a full simulated year, each repeated over three separate world seeds. One of the runs used the Claude Code harness, where the paper reports that most models showed marked improvement in the full-year setting compared with the shorter scenarios.

Pass thresholds are set against a scripted “smart-triage” anchor policy rather than an arbitrary number, and the authors say the reward function was hardened against a catalogue of known reward hacks, with every episode replaying identically given the same sequence of actions.

The numbers

No tested model matched the scripted policy on average. The strongest model, DeepSeek-V4-Pro, passed 49 percent of task-seed cells against the heuristic’s 97 percent, and human experts working the identical tools and budgets scored a mean composite of 0.708 versus 0.700 for the best model, edging out every model tested. In a reinforcement learning run, Qwen3.5-27B trained with GRPO on five disjoint tasks raised its mean composite on held-out evaluation tasks from 0.136 to 0.373, suggesting the gap narrows with targeted post-training even though the full environment and evaluation suite stay withheld to keep the benchmark from being gamed by future training runs.