Paper Reports Station Agents Rediscover 62.7% of Withheld ICLR Findings

Researchers Wenyu Du and Stephen Chung posted a paper to arXiv on October 6, 2026, asking a question that sits behind much of the current enthusiasm for AI agents: can a system find things out on its own, rather than optimize toward a metric somebody else already defined. Their testbed, called Station, is an open-world environment in which several agents operate together to simulate a working scientific ecosystem, exploring, proposing and checking ideas rather than chasing a single score.

To test open-ended discovery honestly, the authors built tasks from three recent oral papers presented at ICLR. Agents received only the main research question each paper set out to answer. The paper’s actual results were withheld, and the agents had no access to the web, so they could not simply look up what the original authors found. The authors then checked how many of the original findings, broken into separate criteria, the agents reconstructed on their own.

Two mechanisms were added to help agents keep exploring when there is no intermediate score telling them whether they are on the right track: a Supervisor mechanism that oversees the research process, and periodic Meta Reflection, which prompts agents to step back and reassess their direction. Ablation tests, which remove one mechanism at a time to see what each contributes, indicated that combining both improved how much ground the agents covered and how well they sustained a line of inquiry rather than drifting.

The authors also ran Station on two further open-ended tasks that had no existing paper to check against, what they call tasks without an oracle. In those cases, some of the conclusions the agents reached closely matched findings that researchers only reported after Station’s training data cutoff, meaning the agents could not have simply memorized the answer.

The numbers

• Station rediscovered 62.7 percent of the withheld findings on average across the three ICLR tasks.
• A comparison system, Codex Multiagent-v2, rediscovered 15.4 percent of the same findings under the identical setup.
• A second comparison system, AI Scientist-v2, scored in the mid-teens to low twenties, well below Station’s result.

The gap between those figures is the paper’s central claim: the two added mechanisms, not simply a stronger underlying model, account for most of the difference, since the comparison systems were working from comparable base capabilities. For anyone trying to judge whether an “AI scientist” headline describes a working method or a cherry-picked demo, this paper is useful precisely because it withheld the answer key rather than letting agents search for it, and because it reports a result against two named baselines instead of a single showcase run. That is a higher bar than most agent papers set for themselves, and it is one that future claims in this area will now be measured against.