Paper Tests Whether Claude Gives Another State's Medicaid Figure by Mistake

A paper posted to arXiv on October 7, 2026, by researcher Jiayu Feng asks a narrow but useful question about state-specific policy answers from large language models: when a model gets a state’s Medicaid income-eligibility figure wrong, is it inventing a number, or is it quietly handing back a real figure that belongs to a different state. The design holds the question wording fixed and only changes which jurisdiction is named, run across all 50 states plus the District of Columbia and three precisely defined eligibility quantities.

The gold values come from an official data book, checked against an independent source and agreeing in all but one of 102 checked cells, which the paper treats as the reference truth against which model answers are scored. Under a pre-registered protocol, the two systems tested, Claude Sonnet 5.5 and GPT-5.6 Sol, each produced the same wrong, other-state answer across two independent repeats for a share of the 153 items: 10 for Claude, 25 for the other model.

The paper’s sharper point is about measurement rather than the raw counts. Simply crediting any wrong answer that happens to match another state’s value produces 3-5x more apparent cross-state substitutions than a stricter check that also looks for the asked state’s own number under a different accounting convention or from an earlier year. Many answers that look like a borrowed figure from a neighbouring state turn out to be the right state’s own historical value, just not the current one. The authors argue that any claim about cross-jurisdiction error needs a complete same-state reference set to rule that out, and say they will release the protocol, the gold table and all model outputs.

The numbers

• 51 jurisdictions were tested: every U.S. state plus the District of Columbia.
• 153 fixed-wording items were scored across three Medicaid income-eligibility quantities.
• Claude Sonnet 5.5 reproducibly returned another state’s current figure on 10 of those items, fewer than the other model tested.
• Naive crediting of any matching wrong answer inflated the apparent substitution count 3-5x over a full same-state check.

The gap between 10 reproducible substitutions and a 3-5x inflation factor is the paper’s real warning: a team auditing a policy chatbot by spot-checking answers against other states’ tables, rather than against the asked state’s own full record across years and conventions, will likely overcount the failure mode it is trying to measure.