Paper Tests ChatGPT, Gemini and Claude on Arctic Science Abstention

Researchers released a paper on October 7, 2026 examining a narrow but practical failure mode in how large language models handle scientific questions they cannot actually answer. The authors built ArcticQA, a set of 194 multiple choice questions drawn from primary Arctic research, with automated checks confirming that each correct answer is genuinely supported by the underlying source evidence and that the wrong answers genuinely contradict it. That groundwork matters because a benchmark testing abstention is only as good as its certainty that the “correct” option really is correct and the distractors really are wrong.

The core contribution is ArcticAbstain, a paired benchmark. Each question appears in two forms: one where the correct answer is present among the options, and one where it has been swapped out for a distractor, with an explicit “abstain” choice offered in both versions. The logic is that a model which abstains constantly regardless of whether the right answer is actually there is not demonstrating judgment, it is just hedging. The paired design is meant to isolate whether a model’s abstention behavior actually tracks answer availability rather than running at some fixed rate.

The team tested eight models from the Gemini, Claude and ChatGPT families, all run at high reasoning effort, with three trials per condition. That produced 9,312 recorded responses in total.

The numbers

Abstention rates when the correct answer was present ranged from 0.0% to 63.0% across the eight models, a spread wide enough to suggest the tested systems disagree fundamentally about when silence is the right call. When the correct answer was replaced with a distractor, abstention rose by 5.05 percentage points on average, a smaller shift than the baseline spread between models.

The authors frame this as evidence of substantial baseline differences between models and argue that abstention frequency and abstention responsiveness need to be evaluated together, since a model can look cautious on one measure while failing the other. For a service like ChatGPT used in research or technical settings, the finding points to a specific risk: a model that rarely refuses to answer is not necessarily more capable, it may simply be less responsive to missing evidence, and a user cannot tell the difference without a test like this one.