IdeaScientist Paper Benchmarks Ideation Agents Against Claude-4.8-Opus

A paper posted to arXiv on October 2, 2026 treats research ideation, the step of proposing a promising, well grounded study direction, as a task worth automating on its own rather than folding it into a broader autoresearch pipeline. The authors, working across Meta and several collaborating institutions, built a system called IdeaScientist that splits the job into three roles: finding gaps in existing work, drawing an innovation from an analogous problem in another field, and writing that intuition up as a full proposal. Each role is trained separately with reinforcement learning rather than prompted once and left alone.

To give the roles something to learn from, and something to be checked against, the team built the Svalbard Idea Vault, a corpus of 2.77M decomposed research ideas used for retrieval, training and what the paper calls temporally controlled evaluation. That control matters here: the evaluation withholds literature published after a cutoff date, then scores how closely the system’s proposed directions match the directions that 15,000 human researchers actually pursued afterward, so the system is judged on foresight rather than on matching known answers.

On a 27B parameter open backbone, IdeaScientist beat the strongest open-source autoresearch baseline by 14.0 percent, which the authors attribute mainly to gains in novelty rather than feasibility. The comparison that will draw more attention outside the paper is the one against commercial tooling: on that same open backbone, IdeaScientist also outperformed Claude Code SDK running Claude-4.8-Opus, and Codex SDK running GPT-5.4, by up to 5.9 percent.

The numbers

• 2.77M decomposed research ideas make up the Svalbard Idea Vault built for training and evaluation.
• IdeaScientist beats the strongest open-source autoresearch baseline by 14.0 percent on a 27B backbone.
• It beats Claude Code SDK paired with Claude-4.8-Opus, and Codex SDK paired with GPT-5.4, by up to 5.9 percent.
• The evaluation checks proposed directions against 15K papers later authored by human researchers.

The headline claim is narrow: one open 27B model, wrapped in a trained three-role pipeline, edged out a frontier model used through a coding harness on one ideation benchmark. It says nothing about writing quality, code generation or the dozens of other tasks where Claude-4.8-Opus is deployed daily, and the paper does not claim otherwise.