Paper Benchmarks Claude Sonnet 4 on Geospatial Tool Use Tasks

A team of six researchers, led by Gabriel Diaz-Ireland, submitted a paper on October 6, 2026 describing GeoNatureAgent, a framework built to check whether a tool-using agent actually picks the right operation when it is pointed at a real geospatial API. The setup fixes a sixteen-tool interface, published as a Model Context Protocol server, so that whichever model is under test is the only variable changing: same tools, same task suite, same deterministic scorer.

The flagship benchmark runs 103 tasks, split into a 93-task main suite spread across 18 categories plus a ten-task comparison set, against an open, self-hostable API covering three environmental indicators in Spain and Portugal. Nine large language models were evaluated across three seeds at temperature 1.0, with the authors deliberately treating capability and per-case cost as separate measures rather than folding them into one score.

Claude Sonnet 4 came out on top, scoring 61.7% across all 103 tasks and 60.8% on the main suite, ahead of DeepSeek V3.2 at 57.9%. No other model in the lineup cleared 53%. The paper also plots a cost-accuracy frontier that is dominated by open-weight models: DeepSeek V3.2 reaches 93% of Claude’s capability while costing 11.3 times less at list price, which is the kind of tradeoff a procurement decision actually has to weigh rather than a single leaderboard number.

A separate finding complicates how those scores should be read against other benchmarks. Under the paper’s strict all-checks scoring, the best model lands 24 to 36 points below the 85 to 97% range typically reported on general-purpose GIS benchmarks. But when partial credit is given per check instead, the top four models cluster at 86 to 90%, which the authors argue shows that much of the apparent gap comes from how strictly a benchmark marks partial progress, not from the geospatial tasks being harder than general GIS work.

The numbers

• Claude Sonnet 4: 61.7% +/- 0.7% across all 103 tasks, 60.8% on the main suite alone.
• DeepSeek V3.2: 57.9% capability, at 11.3 times lower list-price cost than Claude.
• No other evaluated model exceeded 53% capability.

The authors note the server, harness, benchmark and API are public, and that swapping in different tool executors and tasks builds an equivalent benchmark for any other geospatial domain.