New Benchmark Tests Whether AI Agents Can Investigate Causal Systems

A new benchmark called ACED-Bench asks a question that most existing AI evaluations skip over: not whether a model can produce a correct answer, but whether it can work out what evidence it needs to gather before it answers at all. The paper, posted to arXiv on October 9, 2026, builds what its authors call a neuro-symbolic benchmark. A large language model writes the readable surface of each scientific scenario, the prose a person would read, while a hidden probabilistic graphical model underneath fixes the real causal graph, the data-generating process, what each possible intervention actually does, and the correct oracle answer. An agent exploring the scenario sees natural language. The system grading it sees exact structure, which is what lets the benchmark be run at scale and scored without a human in the loop.

The authors tested seven current large language models and two older-generation models across three different agent designs, which is to say three different ways of letting an agent choose what to investigate and when to stop. Two separate test sets make up the evaluation. ACED-QUAL is a smaller qualification check that only asks whether an agent can draw a correct yes or no conclusion once evidence already exists. ACED-CORE is the harder main leaderboard, where the agent has to decide which intervention to run, interpret the result, and commit to an answer rather than continuing to probe indefinitely.

The paper’s evidence-ledger analysis is the part worth reading closely: it traces failures back to specific moments in an agent’s own investigation rather than just scoring a final wrong answer. The authors report agents testing the wrong intervention, losing track of evidence they had already collected, or continuing to probe a system after they already had enough evidence to answer correctly.

The numbers

ACED-CORE holds 180 tasks in total, split into 60 set-valued causal ancestry and descendancy questions and 120 constrained causal-decision tasks. The separate qualification check, ACED-QUAL, adds a further 150-question yes/no test built on the same hidden interface. Nine models in all, seven of them current and two from an earlier generation, were run across three agent designs against that combined set.

For a team deciding whether to let an agent run its own experiments rather than answer a fixed prompt, the paper’s failure analysis is the more useful artifact than the leaderboard score itself, since it shows where an agent’s own bookkeeping, not its reasoning, breaks down first.