Researchers Detail Sparse Attention Method Tested on Nvidia B200

The paper addresses a cost that shows up whenever a language model has to hold a long conversation or document in context: the key-value cache that decoding reads from grows with every token, and reading it becomes the main thing slowing generation down. Sparse attention tries to cut that cost by skipping over tokens that matter less to the next prediction, but the authors argue most existing methods pick which tokens to skip in blocks that are too coarse, or they pick individual tokens in a way that loses the structure needed to rank them accurately.

Their method, called Joint Subspace Sparse Attention, learns a shared low-dimensional space for queries and keys that preserves token-level scoring accuracy, solved with a closed-form spectral approach rather than an iterative one. The authors state plainly that trying to refine the harder, non-convex version of this problem directly gave no measurable improvement in their tests, which is a useful admission: it says the simpler, tractable approximation was not a compromise they settled for, but one they checked and found sufficient.

The second half of the work is about execution rather than selection. Picking scattered tokens from memory is itself slow if the hardware has to gather them one at a time, so the team built what they call a gather-free, token-level indexed attention kernel that works with the grouped-query attention layout most production models already use. They tested the combined method across three different attention architectures, including models using multi-head latent attention, and report that it stays close to full-attention accuracy in each case while avoiding the generation-length inflation that competing sparse methods showed in their tests.

The numbers

Run on an Nvidia B200 GPU at a 128K token context length, the method reached up to 6.43x the decoding throughput of full attention. At the scale of a 700 billion parameter model, the authors report the method reaches 96 to 99 percent of the decoding throughput of a sparse attention mechanism built directly into the model’s own architecture, while matching that native method’s reasoning accuracy without any additional training.

That last comparison is the one worth sitting with. A sparse attention mechanism trained into a model from the start has an obvious advantage: the model learned around it. A method that reaches most of that speed without touching the model’s weights is a tool that could, in principle, be applied after the fact to models already in production, which matters more to a team running inference at scale than to one training a new model from scratch.