Independent Audit Finds Cheap Moderation Model Rivals Claude

Content moderation at the scale of billions of posts cannot afford to run a frontier model on every item, which is the problem a new paper takes on directly. Published on October 23, 2026 by researcher Karan Yadav, the work runs what it calls the first independent audit of a frozen “System One” moderation model, a system that returns typed, calibrated decisions rather than text, priced at roughly USD 0.000025 per call and answering in around 100 milliseconds. The audit tests it against 14 frontier reasoning models on a shared probe set, including Anthropic’s Claude alongside the GPT-5.x, Gemini 3 and Grok families, over 142,658 human-labeled comments.

The headline result is that the cheap model holds its own on raw accuracy. Its zero-shot classification score, an AUROC of 0.83, could not be statistically separated from 13 of the 14 frontier systems in a paired bootstrap test, a result the author is careful to call a failure to reject parity rather than proof the systems are equal. The cheap model also matched or beat several of the larger systems on calibration, while costing one to three orders of magnitude less to run.

The second finding complicates the first. The paper argues that “confidence” in these systems actually conflates two separate things: human disagreement, which shows up on satire and coded language where annotators themselves split, and decision risk, the cost of an automated action going wrong. The paper says these call for opposite escalation signals, so a single confidence number cannot serve both. Splitting a judgment into parallel sub-questions can recover a disagreement signal the model’s own confidence misses, but the paper notes this fix is narrow: it helps only the weakest, disagreement-blind models, not a specialist toxicity model or the strongest large language models, and its benefit shifts with the data it is tested on.

The numbers

• 142,658 human-labeled comments formed the audit set.
• 14 frontier reasoning models, spanning the GPT-5.x, Claude, Gemini 3 and Grok families, were benchmarked on the same probe set.
• The cheap model’s zero-shot AUROC came in at 0.83.
• Each call to the cheap model cost roughly USD 0.000025, against the price of running a full reasoning model on the same post.

The author plans to release the instrument, the cached runs and the analysis, which would let other moderation teams check the comparison against their own data rather than take the headline number on faith.