GAVEL Paper Proposes LLM Judge to Adjudicate Clinical Timeline Extraction Disputes

GAVEL, short for Grounded Adjudication of Variations across Extracted TimeLines, is a protocol described in a paper submitted to arXiv on September 11, 2026, and accepted for the AMIA Informatics Summit 2027. Instead of scoring an extraction pipeline against one expert-written reference timeline, the method has a large language model act as a judge: it reads the original case report alongside two competing timelines and returns a discrepancy type, a verdict, and the exact passage in the report that supports its ruling for every difference it finds.

The authors tested the judge’s own event matcher first, since a comparison protocol is only as good as its ability to line up the same clinical event across two differently structured timelines. They then applied GAVEL to outputs from GPT5.6sol and DeepSeek V3.2, along with timelines from other extractors and from human annotators, reading and confirming the judge’s rulings by hand rather than trusting them outright.

The numbers

The authors reviewed the judge’s output on 2,738 findings drawn from GPT5.6sol and DeepSeek V3.2 timelines, then had people confirm 89.4% and 88.6% of those rulings on manual check, a gap that shows the judge is usable but not infallible. A confidence threshold set at 0.10 split the matcher’s true match rate into 60% below the cutoff and 48% above it, evidence the boundary itself needs tuning rather than treating any single cutoff as settled. Across 126 case reports, timelines that GAVEL helped merge were preferred over either original in 77.0% of comparisons, with a 95% confidence interval running from 69.8% to 84.1%. Merging also cut discrepancies attributed to the evaluated timeline from 7.63 per report down to 0.85.

That drop matters for a specific workflow choice: teams building clinical timeline extractors currently have to pick one output and accept whatever errors its single reference annotation missed. GAVEL’s report-grounded merging offers a way to combine two candidate timelines, including one built on DeepSeek V3.2, into something closer to what the case report actually supports, without ever declaring either input the ground truth. The paper frames this as a general evaluation and revision tool rather than a one-off benchmark result, which is the more consequential claim for anyone deciding whether to adopt an LLM-judge step ahead of manual chart review.