Paper Finds Agent Harnesses Do Not Reliably Improve Reasoning Accuracy

Researchers led by Zhanke Zhou and eighteen co-authors published a preprint describing AlphaDiana, a measurement system built to answer a question that agent builders usually assume rather than test: does wrapping a reasoning model in a harness, adding tools, loading skill instructions or retaining memory actually make it more accurate. The paper, posted to arXiv on October 14, 2026, argues that the answer is not a simple yes.

AlphaDiana separates the parts of an agent that are usually bundled together. It keeps each harness’s own control loop intact while giving researchers modular interfaces to swap datasets, environments, scorers and agent components independently, so a change in memory retention can be measured without also changing file reuse or tool access. The system supports both macro comparisons of complete agents and micro interventions on a single component, under matched conditions, which the authors say is needed to tell whether a gain came from the harness itself or from an unrelated change in setup.

The headline result is that harnesses do not consistently raise accuracy. On some questions a model could not answer alone, a harness produced a correct answer. But on other questions the same model already answered correctly without any tools, and the harness introduced a wrong answer instead. Across the benchmarks tested, those losses sometimes outweighed the gains, so a harness that looks like a net improvement on one measure can be a net loss once both directions are counted together. The same pattern held for narrower interventions: neither adding skill guidance nor retaining conversation history reliably improved accuracy on its own.

The authors also flag a measurement trap in how teams read agent behavior. Low token entropy, often read as a sign of a confident, well-calibrated agent, can still accompany a wrong answer. And when accuracy falls, an agent can produce fewer output tokens overall while needing more output tokens for each answer it actually gets right, so a shrinking token budget does not necessarily mean a cheaper correct answer.

The numbers

The study evaluated three open-source reasoning models against three open-source agent harnesses, run across five answer-based benchmarks. Two of those models were tested further on two interactive benchmarks, giving the authors both static question-answering tasks and settings where the agent has to act over several turns. AlphaDiana logs correctness and execution status alongside full traces and resource records for every run, which is what lets the paper compare harnesses and diagnose failures under matched task and resource conditions rather than from end results alone.