Paper Finds Claude Opus 5 Favors Biology Over Logic in Prose Reports

Anthropic’s Claude Opus 5 was one of two models tested in a study posted to arXiv on September 29, 2026, asking whether a language model resolves scientific disagreements by weighing stated constraints or by falling back on what it expects biology to look like. The researchers built an unsatisfiable set of XOR constraints, then wrote those constraints up as scientific reports from different laboratories, so the model had to decide which lab’s numbers to trust. When the constraints were handed over directly, both models did well: Claude Opus 5 recovered the better-supported assignment 96% of the time and GPT-5.6 Sol managed 90%.

The gap opened once the same constraints were buried inside ordinary scientific prose instead of listed plainly. In that setting Claude Opus 5 began reaching for the answer that matched textbook biology rather than the answer the numbers actually supported. Stripping out the biological framing raised its recovery of the correct assignment from 27% to 79%, a change the authors report as statistically significant. Adding a request to formalize the problem alongside an explicit note that the two reports described a paired design pushed recovery to 92%. GPT-5.6 Sol moved far less across the same changes, and neither adjustment reached significance for that model.

The paper, accepted at a NeurIPS 2026 workshop on verification in AI science, reads as a caution for anyone using a model to check whether two findings from different sources genuinely conflict. The failure is not arithmetic. Claude Opus 5 can do the comparison correctly when asked directly, so the trouble sits earlier, in how it decides whether two reports describe the same kind of measurement once that decision is dressed in natural language rather than stated as a rule.

The numbers

• 96%: Claude Opus 5’s recovery rate when constraints were stated directly.
• 90%: GPT-5.6 Sol’s recovery rate under the same direct condition.
• 27%: Claude Opus 5’s recovery rate in prose form before the biological framing was removed, rising to 79% once it was stripped out.
• 92%: recovery once a formalization request and a paired-design cue were added on top of that fix.

A team that already pays Anthropic’s published rate for Claude Opus 5 is not getting this reliability for free. Moving from the 27% baseline to 92% needs an extra formalization step and an explicit paired-design prompt on every query, turning what looked like a single call into a short pipeline before the output can be trusted for a genuine literature check.