New Offline Benchmark Study Shows Fine-Tuned Model Beating Claude Opus 4.8

A group of researchers, Chengguang Gan, Yunhao Liang, QingHao Zhang and Shiwen Ni, published a paper on September 23, 2026, describing a new way to test web browsing agents without the drift that live environments introduce. Their argument is that scoring agents in a live browser is unreliable because the page state and the judge model change between runs, so the same checkpoint can score differently each time it is tested. Their fix, called WebMRE, replays 541 tasks and 5,293 steps taken from successful WebArena trajectories offline, with labels checked by hand, so a checkpoint gets the identical score on every run.

Each step in the benchmark pairs a plain language guide sentence, the kind a person would write to explain what to click next, with the grounded action a model actually needs to take. That pairing let the team study what they call a mutual reinforcement effect between the instruction and the action. Decoding a guide alongside the action raised element selection accuracy over an action-only baseline, and the size of that lift grew with model scale, from small gains on a 4 billion parameter Qwen3.5 model to larger gains on a 9 billion parameter version.

A follow-up test went further: forcing the correct guide sentence as a decoding prefix raised action accuracy sharply, while forcing a mismatched guide from a different step collapsed it near zero, and a reworded version of the correct guide still recovered about half the gain. That pattern is the paper’s evidence that the guide text carries real instruction meaning rather than just repeating the name of the target element.

The authors also report that models they fine-tuned using these findings beat GPT-5.5, Claude Opus 4.8 and Gemini 3.5 Flash, all tested zero shot with no extra tuning, across every metric the offline benchmark measures.

The numbers

The benchmark totals 541 tasks and 5,293 steps. On the mediation test, forcing the correct guide sentence as a decoding prefix raised action accuracy from .422 to .684, while swapping in another step’s guide dragged accuracy down to .055, showing how much of the model’s performance rests on reading the instruction rather than guessing from the screen alone.