Preprint Maps When Muon's Orthogonal Update Beats Plain Gradient Descent

The paper’s core move is to strip away learning rate tuning by comparing update directions under line search, then study PolarGD, the orthogonalized step that sits inside the Muon optimizer, on matrix quadratic problems where the whole trajectory can be written out in closed form. That setting lets the authors say precisely when orthogonalizing a gradient update helps and when it does not, rather than relying on the usual anecdotal reports that Muon trains some large language models faster than AdamW.

Their central result is a one step convergence guarantee: PolarGD reaches the minimizer in a single step whenever the rows of the error are orthogonal and have equal norms, regardless of what the Hessian looks like. Plain gradient descent only gets that same one step guarantee when every eigenvalue of the Hessian is equal, a far narrower condition. The authors also show PolarGD, unlike sign descent, is unaffected by rotating the problem, which rules out an entire class of coordinate tricks as an explanation for why the method works.

Over many steps the picture changes. The paper shows PolarGD behaves like gradient descent preconditioned by the current iterate, so its speed depends on how well conditioned that iterate is rather than on the Hessian’s conditioning. That framing explains both the fast regimes practitioners report and a slow regime the authors identify, caused by tiny singular values in the iterate, where PolarGD can run much slower than plain gradient descent.

The numbers

The preprint runs 47 pages and carries 13 figures, and traces the slow regime back to the Newton-Schulz iteration Muon actually uses in practice. Because Muon runs only a few Newton-Schulz steps rather than exact orthogonalization, it treats small singular values the same way gradient descent does, so fewer iterations can sometimes converge faster than full orthogonalization would. Checking a transformer trained with Muon, the authors find about a quarter of the singular values in its attention matrices fall into this regime, which is the figure likely to draw the most attention from teams running Muon at scale.

A shorter version of the work was presented at the ICML 2026 Workshop on High-dimensional Learning Dynamics, and the full preprint was submitted to arXiv on October 21, 2026 by Sam Laing on behalf of the coauthors.