Paper Tests Whether DeepSeek Models Earn Predictive Credit for Explanations

Researchers behind a new paper, posted to arXiv on 29 September 2026, set out to measure what a scientific explanation adds to an experimental forecast once a research agent has already stated what it plans to test. The design pairs two forecasts that share the same intervention, the same forecaster and the same outcome, but vary whether the agent gives a plain description, a matched explanation, or extra donor context, then runs five separate checks for commitment, delivery, predictive gain, alignment and whether the agent used signals it already knew about.

The test set spans 336 prospective states drawn from controlled learning tasks, 12 Tox21 toxicology endpoints and 24 OpenML benchmark tasks. DeepSeek V4 Pro was the requested model for the main Tox21 runs, where matched and donor explanation cards reduced secondary drift against the preregistered harm test. A separate DeepSeek V4 Flash replay of the matched condition moved point mean absolute error in the wrong direction and still missed the interval score equivalence test against the donor condition, while giving every one of the 144 direct-text notes with no detectable gain in point accuracy over a plain description.

Across the full protocol the frozen credit decision the authors set out to reach stayed inconclusive. The Tox21 harm test and the OpenML joint formation, point equivalence and repeatability rule were both unmet, and the seed-donor intervals for both benchmark sets spanned zero, meaning the paper cannot say with confidence that an explanation adds predictive value beyond description at the donor resolutions it tested.

The numbers

Under DeepSeek V4 Pro, matched and donor explanation cards cut secondary Tox21 drift by 64.5 and 59.1 percent against the harm test baseline. Swapping in DeepSeek V4 Flash for a matched-condition replay moved point MAE from .01823 to .02020, a step in the wrong direction that also missed the donor equivalence bar. On OpenML, full-card assignment widened nominal 80 percent intervals by 21 percent, with 49.3 percent coverage against 51.4 percent for plain description.

For a reader deciding which model to route an evaluation workload through, the gap between the two DeepSeek models is the finding worth noticing. DeepSeek’s own API price table lists V4 Pro at $3.96 per million output tokens at peak hours against $1.32 for V4 Flash, a saving of exactly the size that would tempt a team to switch, but this paper’s own replay shows that switch adding measurable error rather than removing it. A cheaper model to call is not the same as a cheaper model to trust, and the paper gives no sign the gap closes once explanations rather than plain descriptions are what is being scored.