Study Finds Precedent Labels Sway Claude Haiku Approval Calls

A paper posted to arXiv on October 31, 2026, tests how large language models decide refund and claim requests once they are shown a handful of past cases alongside a written policy. The authors, a team of seven researchers, split each past case into two pieces: the facts of what happened, and the decision that was recorded for it. By swapping one piece while holding the request itself fixed, they could isolate which part of a precedent actually moves a model’s judgment.

The study picked genuinely borderline requests. Without any past cases to look at, DeepSeek V4.1 Flash approved the same request on some tries and denied it on others across four attempts, which is the definition of a close call the researchers used to build their test set. Three models were then run against that set with precedents attached: DeepSeek V4.1 Flash, GPT-5.6 Luna and Claude Haiku 5.5.

The labels on near-copy precedents turned out to dominate. When the past cases were close variants of the request itself, flipping the recorded outcome from denial to approval raised the approval rate by roughly 30 to 55 percentage points across the named models, both on policies the authors wrote for the study and, for DeepSeek and Luna, on nine public policies paraphrased from real ones. The facts moved the decision in the opposite direction: shown without any attached outcome, past cases whose facts supported approval made Claude Haiku 5.5 and DeepSeek approve less often than past cases whose facts supported denial, as if the request looked weaker standing next to a strong case.

The numbers

The paper reports that this facts effect was not simply the model correcting toward the right answer. In an added DeepSeek analysis, it showed a similarly sized pull on requests that human annotators had separately judged approvable and on those judged deniable. A further DeepSeek run tested what happens when the near-copy precedents actually belong to a different request: the label-switching effect fell to 9 percentage points from 43, while the facts effect held at a 22 point gap between approval-supporting and denial-supporting facts.

The authors conclude that any system retrieving past decisions as context should test the labels and the facts of its precedent store separately, since the two push a model’s judgment in opposite directions and by very different amounts depending on how closely the retrieved case actually matches the request in front of it.