Paper Proposes Jury Model To Rate AI Ensemble Reliability

Researchers from a four-person team posted a paper to arXiv on October 15, 2026 that reframes how a group of language models should be combined. Instead of treating every model’s vote the same way, or fixing each model’s reputation ahead of time from labelled data, the paper models the group as a jury: each question has one true answer, every expert reports a guess with some unknown reliability, and that reliability is estimated purely from how often each expert’s answers line up with the others. The method uses Expectation Maximisation to pull those reliability scores out of unlabelled answer patterns alone, then folds them into what the authors show reduces to a log odds weighted linear score across the experts’ answer distributions.

The authors tested the idea on a benchmark of 1560 questions, pairing nine experts with up to 3B parameters each. The jury’s label-free scoring performed comparably to the same fusion rule given label-fitted weights, meaning the paper found no statistically significant benefit from having access to ground truth labels for calibrating the panel. The jury version also beat the best single model when that model was given matched self-consistency sampling, and it beat the strongest label-free alternative method across the full question set.

The numbers

The benchmark ran to 1560 questions, large enough to let the reliability estimates settle rather than resting on a handful of trial questions. The tested jury used nine experts, each no larger than 3B parameters, which is a modest size by current standards and suggests the gains come from how the panel is scored rather than from throwing bigger models at the problem. The paper itself runs 35 pages with four figures and 13 tables in its appendices, reflecting the breadth of the ablations the authors ran to check the method against labelled alternatives.

For anyone assembling a panel of smaller, cheaper models to check each other rather than running one large model alone, the result suggests that collecting labelled validation data to calibrate trust weights may be unnecessary overhead. A team could instead let the panel score itself from its own pattern of agreement, cutting a data collection step out of a workflow that otherwise needs it before any fusion rule can be tuned with confidence. That is a different kind of saving than a lower bill or a higher rate limit, but it is a real one: fewer labelled examples to source, annotate and maintain before a multi-model review process can be trusted at all.