Paper Finds Arena-Hard Judge Verdicts Shift Even When Scores Hold Steady

The paper, by Louise Yorke and accepted at the JUDGe workshop at NeurIPS 2026, asks a narrow but uncomfortable question about a benchmarking practice that has become routine: when an LLM is asked to pick the better of two answers, does the same judge give the same verdict twice, and does that verdict depend on how the instructions are worded. Both questions sit underneath every leaderboard that reports an LLM-judged win rate as if it were a stable measurement.

The study holds the two models being compared fixed and runs GPT-4.1 as the judge over 500 prompts from Arena-Hard v2.0, a pinned model version at temperature zero. It first reruns the exact same setup unchanged, then builds eight combinations of judging criteria, output format and task framing to see how much the instructions themselves move the result. Repeating the unchanged setup still moved the aggregate score by about one point, a gap small enough to look like noise, yet the underlying win, tie or loss call changed on a sizeable share of individual prompts. Changing the instructions pushed that further: one variant that asked for a verdict with no reasoning and trimmed several other instructions produced far more outcome changes than the unchanged reruns did on average, alongside a real fall in the aggregate score. A separate variant that told the judge to weight accuracy moved almost as many individual outcomes while leaving the aggregate score close to flat, which is the paper’s central warning: a steady headline number does not mean the verdicts behind it are steady.

The paper’s own recommendation is blunt. Anyone who relies on individual LLM-judge verdicts, whether as preference labels for training or as a way to score one response, should rerun each setup they use and report prompt level agreement alongside the aggregate score, rather than treating one run of one prompt set as settled.

The numbers

• 21.2% of prompts changed win, tie or loss outcome when the identical default setup was rerun unchanged.
• 500 Arena-Hard v2.0 prompts were judged, with the two compared models’ answers held fixed across every setup.
• 28.2 percentage points more prompts changed outcome in the setup that dropped reasoning and other instructions, compared with the unchanged reruns.
• 4.09 points was the aggregate score fall in that same reasoning-dropped setup.

The practical lesson for anyone citing an Arena-Hard style score is to treat the published number as one sample from a noisy process, not a fixed property of the model being judged, and to ask whether the people who produced it ever checked a rerun.