Paper Tests Claude Haiku 4.5 on Legal Clause Drafting Risk

A paper posted to arXiv on 22 August 2026 sets out a framework for grading how large language models draft contract clauses, arguing that fluency and accuracy scores miss the failures that matter to a lawyer. The authors, Devansh Singh and Sundaraparipurnan Narayanan, note that a clause can read smoothly while still omitting an essential carve-out, allocating risk in a way that would not hold up, assuming the wrong jurisdiction, or leaving a party exposed to a regulator. The work has been accepted to the AI for Law Workshop at ICML 2026.

The study runs four models, Claude Haiku 4.5, Gemini 2.5 Flash Lite, GPT 5.4 Nano and Qwen 3.5 Flash, through 22 clause categories and 34 legally-motivated failure modes. It combines two evaluation tools: CLAUSE, which sorts prompts by legal function and by the kind of failure they are meant to expose, and LENS-CRAFT, which scores each output across nine legal-quality dimensions. Rather than averaging those nine scores into one number, the paper applies what it calls a Max Severity Principle, so that one legally decisive defect stays visible instead of being smoothed away by otherwise strong writing.

The paper is a protocol and taxonomy paper rather than a leaderboard. It publishes the evaluation method, the failure-mode taxonomy, an analysis plan and a results structure, and argues that legal AI evaluation generally needs to move away from aggregate accuracy toward assessment that is specific to the clause type and sensitive to risk.

The numbers

The test set spans 22 contract clause categories and checks outputs against 34 legally-motivated failure modes, a wider net than a single pass-fail accuracy score would catch. Claude Haiku 4.5 is one of four models scored under that net, sitting alongside Gemini, GPT and Qwen variants rather than being singled out. On Anthropic’s own pricing page, the model runs at $1 per million input tokens and $5 per million output tokens, among the cheaper entries in the current Claude line-up.

That price point is what makes the paper’s warning practical rather than academic. A cheap model is an easy default for high-volume clause drafting, and the paper’s severity-first scoring is built precisely to catch the case where a low-cost draft looks finished but hides one clause that would not survive a dispute.