OR-License Benchmark Tests Claude Code On Real Consulting Work

A group of researchers released OR-License, a benchmark meant to close a gap in how operations research agents get graded. Most existing tests start from a tidy, already-formulated optimization problem. OR-License instead starts from the messy originals: 31 instances across 10 industrial scenarios, each one a de-identified paid consulting engagement with the client’s original requirements, raw input data and deliverable format left intact. The engagements behind the benchmark were together worth USD 3.68 million, which the authors use to argue the work is representative of what a real OR shop actually sells, not a textbook abstraction of it.

To stress the hardest part of that work, the authors built ORSim, a simulated client that will disclose business facts on request but will not hand over any operations research expertise. An agent has to ask the right questions, at the right stage, to get what it needs. Eleven frontier agents were run against the benchmark, among them a configuration built on Claude Code. Several agents produced individual solutions whose objective quality beat the expert reference once a solution was feasible at all. None matched the expert on the benchmark’s combined Deployment-Adjusted Quality score, which counts an infeasible delivery as a zero rather than ignoring it.

The strongest Claude Code configuration reached 88.39% strict feasibility and a Deployment-Adjusted Quality of 0.87 against the expert’s reference score of 1. The authors trace the shortfall to two separate failure types: agents mis-formulating the underlying problem, and agents implementing a correct formulation incorrectly. They also found that confirming information with the simulated client kept mattering well past the opening clarification stage, continuing into solution development and validation.

The numbers

• USD 3.68 million: the combined value of the paid engagements the 31 benchmark instances are drawn from.
• 88.39%: the strict feasibility rate of the strongest Claude Code configuration tested.
• 0.87: that configuration’s Deployment-Adjusted Quality score, against an expert reference of 1.
• 31: the number of end-to-end industrial scenarios the benchmark preserves after de-identification.

For a reader weighing whether to route a consulting-style workload to an agent, the finding to hold onto is the distance between a strong per-task number and a weaker delivered-package score, since a client paying for a finished answer is paying for the package, not the best single result inside it.