New MathVis Benchmark Tests Claude and Rivals on Diagram Reasoning

A team of six researchers, led by Jingjing Wei, released MathVis on October 9, 2026, a benchmark built to answer a narrow question: can a model reason by drawing, not just by writing symbols. Each of the 512 tasks pairs a mathematical statement with a reference diagram, executable TikZ or SVG code, a Gold Intermediate Representation, atomic rubrics and error codes, so a grader can check whether a generated figure actually carries the reasoning a human mathematician would put on paper.

The headline numbers are modest across the board. GPT-6-Astra led with a pass@1 score of 31.25%, Claude Fable 5 followed at 26.56%, and Gemini 3.1 Pro Preview came in at 22.85%. None of the three cleared a third of the tasks on a first attempt, which the paper treats as evidence that visual reasoning on frontier mathematics remains a genuinely hard problem for current systems rather than a gap that will close with a minor prompt change.

What makes the result more pointed than a simple leaderboard is the explanation the authors give for the gap. They report that even when a model’s output technically contains every element a correct diagram needs, the LLMs tested lean on formal languages, symbols and graphics-program syntax rather than producing diagrams that mediate the reasoning step by step. The paper’s own phrase is that solutions are “often visually confusing to humans”, meaning a figure can pass an automated rubric while still failing the actual test a diagram is meant to serve, which is helping a person see the proof.

That distinction matters for anyone building on top of Claude, GPT or Gemini for education or technical writing tools. A model that writes correct TikZ code is not the same as a model that produces a diagram a student can read, and MathVis is designed specifically to catch that difference with its Gold Intermediate Representation and atomic rubrics rather than only checking final answer correctness.

The numbers

The benchmark’s 512 tasks span frontier mathematical statements requiring diagram-assisted reasoning. GPT-6-Astra scored 31.25% pass@1, Claude Fable 5 scored 26.56%, and Gemini 3.1 Pro Preview scored 22.85%, the three figures the paper reports from its evaluation.

The authors frame MathVis as a starting point rather than a finished scoreboard, saying the benchmark is meant to promote further work on letting models simplify their own reasoning through visual diagrams, an aim that puts the burden on future model releases to close a gap this paper has now measured in detail.