Benchmark From Open Chip Tapeouts Shows Claude Models Refusing Sizing Tasks

Researchers Yueting Li and Weihang Ding released SGAnalog, a benchmark built entirely from human-designed analog circuits that have already been manufactured through the open-source Tiny Tapeout program. Rather than relying on textbook circuits a model may have memorized during training, the authors pulled 273 topologically distinct designs at the exact revision recorded for each chip’s manufacturing submission, then processed every one through a fixed containerized pipeline that exports both a schematic image and a matching SPICE netlist from the same source file.

The benchmark splits evaluation into two tasks: reading a schematic and turning it into a netlist, and then proposing the component sizes an author’s own testbench would accept. Across seven models tested on 66 transcription tasks, the strongest system reached 56.1 percent exact graph isomorphism, and six of the seven models dropped sharply once circuits moved from the small to the medium size tier. On 17 separate sizing tasks, the leading model converged on proposals for every one and scored 91.2 out of 100 against the human reference.

The detail most relevant to anyone deciding which model to route a hardware task through is what happened with Claude. The two newest Claude models in the test refused 4 and 11 of the same sizing prompts they had transcribed without objection moments earlier. The authors describe this as a policy limit rather than a capability limit in that one model family, since a proposal left blank scores zero regardless of whether the model could have produced sizes.

The numbers

• 273 topologically distinct top-level designs make up the benchmark, each tied to a recorded Tiny Tapeout shuttle submission.
• The strongest model reached 56.1 percent exact graph isomorphism on the 66 transcription tasks, with six of seven models falling off sharply at the medium size tier.
• The leading model converged on all 17 sizing proposals and scored 91.2 out of 100 against the human reference.
• One of the two newest Claude models refused 11 of the 17 sizing prompts it had transcribed without objection.

The authors also tested what happens when author-chosen labels are stripped from a schematic before a frontier model reads it: exact matches fell, but aggregate structural F1 held, suggesting labels help a model trace connectivity rather than recognize the circuit outright. That separation between transcription skill and sizing willingness is the paper’s central finding, and it is the kind of distinction a benchmark built from memorized textbook circuits could not have surfaced.