Paper Introduces AgentHorizon Benchmark for AI Judges Grading Computer-Use Agents

Computer-use agents, the systems that click, type and navigate across applications on a person’s behalf, are increasingly graded by other AI models rather than by humans, both during training and when checking whether a task actually succeeded. A paper posted to arXiv on October 8, 2026 argues that this shortcut is shakier than it looks once a task spans many steps and multiple applications, because a trajectory can look finished in its final screenshot while quietly breaking a constraint from the original instruction or introducing a side effect nobody wanted.

To test this, the authors built AgentHorizon, drawing on 166 hours of human-recorded trajectories across three operating systems to assemble 1,373 instruction-trajectory pairs. The design trick is pairing closely related instructions and swapping them between trajectories, which manufactures negative examples: a judge is shown a trajectory that completed a similar, but subtly incompatible, request, and has to notice the mismatch rather than reward surface completion. The benchmark ships in three splits, a frontier split, a simplified split, and a development split, so researchers can separate harder cases from easier ones.

Eleven judges were tested two ways: reading a full trajectory of up to 300 screenshots and actions directly, and acting as a coding agent across five separate agent harnesses. The strongest performer, GPT-5.5, reached 80.9% balanced accuracy on the harder AH subset, a result the paper frames as the ceiling rather than a solved problem. Giving judges tool access helped some models but made open-weight models worse, and the paper reports that judges varied sharply in how well they could both accept a genuinely successful trajectory and reject a failed one, rather than defaulting to approval.

The numbers

• 1,373 instruction-trajectory pairs make up the benchmark, drawn from 166 hours of recorded human trajectories.
• 300 screenshots and actions is the longest trajectory passed directly to a judge for evaluation.
• 80.9% balanced accuracy is the best score recorded, by GPT-5.5, on the AH subset.
• Eleven judges were tested across five agent harnesses, with tool-use improving some models and hurting open-weight ones.

The authors conclude that judges still need better ways of locating and verifying evidence hidden deep inside long interaction histories, a gap that matters for anyone deciding how much automated grading to trust before a human review step is removed from a pipeline.