Paper Finds Claude Opus 5.5 Stays Coherent on Hidden-Variable Probability Tests

The paper, titled “Latent Coherence: Do Frontier Language Models’ Probability Judgments Obey the Law of Total Probability?”, does not test whether a model can answer a probability question correctly. It tests whether a model’s own answers agree with each other once a hidden variable sits between the question and the evidence. The setup asks a model, in separate single-turn calls, for the overall probability of an outcome, for the probability of each possible hidden state, and for the probability of the outcome given each of those states. Basic probability theory says the three numbers must combine into the first one; a model that has not been trained for this consistency has no reason to actually produce numbers that do.

Across seven models tested, Claude Opus 5.5 placed second only to GPT-6.1 Sol, with a mean absolute residual of 0.12 log-odds against GPT-6.1 Sol’s 0.08. Claude Haiku 5.5 sat with Gemini 3.8 Flash and DeepSeek V4-Pro among the models showing persistent incoherence, in the range the paper describes as 0.27 to 0.35 log-odds, and the paper reports that this incoherence tends to be specific to particular questions rather than a uniform offset applied everywhere. The same paper ran a causal variant of the test, applying the identical logic to interventional probabilities and the back-door adjustment, with a similar split between the stronger and weaker models.

One further finding stands out for anyone deciding how much to trust a model’s stated confidence. Within each model, items with larger residuals also tended to carry larger errors against the true probability, a correlation the paper measures with a Spearman coefficient between 0.40 and 0.62. That means the residual itself, computed with no access to ground truth, can flag which answers a model gave are less reliable. The paper also found that how much a model reasoned before answering did not predict whether it stayed coherent, either across different models or across different wordings of the same question put to one model.

The numbers

• Claude Opus 5.5’s mean absolute residual: 0.12 log-odds, the second-lowest of the seven models tested.
• GPT-6.1 Sol’s mean absolute residual: 0.08 log-odds, the lowest recorded in the study.