Paper Tests Polynomial Shortcuts For Nvidia GPU Transcendentals

A paper posted to arXiv on September 3, 2026 asks a narrow question about Nvidia’s newest training hardware: can a short polynomial formula, hand tuned and packed into fused multiply add instructions, do the job of the native sigmoid, tanh and SiLU functions faster than the chip’s own special-function unit. The author, Robert Hu, frames the work as a follow-on to FlashAttention-4, which had already shown that attention kernels on Blackwell hit a special-function bottleneck rather than a matrix-math one as GPU generations scale their pipelines at different rates.

The testing ran in two stages. First came an isolated sweep in FP16, comparing native PyTorch evaluation against the packed polynomial programs across working sets that fit in L2 cache and working sets that spill to high-bandwidth memory. Then came integration: the polynomial substitutes, written in BF16 at degree 3 or degree 4, were dropped into four real GB200 training tasks, dense SiLU, tanh-softcapped attention, sigmoid attention and routed-expert SwiGLU, the layer type used inside mixture-of-experts models.

The paper does not stop at speed. It also ran same-checkpoint ablations and one paired pre-training comparison for each of the four tasks, tracking whether swapping the math changes what the model learns. At a common horizon near 100 billion tokens, the smoothed training-loss difference between the polynomial and native versions ranged from a small negative shift to a small positive one across the four tasks, which the author reads as the substitution leaving model behavior effectively unchanged while the clock keeps moving faster.

The numbers

In the isolated FP16 sweep, the polynomial path improved performance between 1.19 and 2.19 times in L2-resident cases and between 1.00 and 1.70 times when the working set lived in high-bandwidth memory. Inside full GB200 training runs, the routed-expert SwiGLU substitution gave the largest complete training-step throughput gain, 8.0%, ahead of tanh-softcapped attention at 2.9% and dense SiLU at 2.7%. The sigmoid-attention swap raised complete-attention forward throughput by 7.4%, though that narrowed to 0.3% once measured across the whole GPU step rather than attention alone.

None of this changes a price list or a quota a reader of this site tracks directly. What it points to is that gains on fixed Nvidia hardware are still being found in software, through kernel-level substitutions rather than new silicon, which matters for anyone deciding whether the next speed-up in their training bill comes from a newer GPU generation or from a library update on the one they already rent.