Paper Tests 50 Million Token Memory Layer on One Nvidia H100

A paper submitted to arXiv on October 7, 2026, tests whether a model needs to hold everything in its context window to use it, or whether saved state can stand in for recomputation. The author, Sietse Schelpe, built galahad-kv, a package that writes each block of a model’s key-value cache, about 16,000 tokens at a time, to encrypted local NVMe disk and reads it back byte exact later instead of running the block through the model again.

The test ran 50,000,000 tokens of real public text through vLLM on a single Nvidia H100, with Gemma 4 12B and Gemma 4 31B taking turns. Every block the authors checked came back from disk with no recomputation at all, at depths running from the start of the stream to the full 50 million tokens, on both model sizes. Loading a saved block beat recomputing it on both speed and energy, and GPU memory held steady for the entire run rather than climbing as the stream grew, which is the behavior that normally forces a long session to be cut short or moved to a bigger card.

The paper is careful about what this is not. It is reuse of stored state rather than a genuinely wider attention window: one block loads at a time, and how well a question gets answered still depends on the model answering it. Gemma 4 31B recovered facts planted deep in the stream far more reliably than the 12B model, and the paper reports that neither model invented an answer when probed. Writing the memory in the first place carries a one-time cost, and the resulting store runs to terabytes of local disk, which is not a free trade so much as a different one.

The numbers

The 50,000,000-token run was served from one Nvidia H100. Reloading a saved block ran 2.8x to 4.3x faster than recomputing it, and used 8.8x to 12.3x less GPU energy, across both model sizes tested. Asked about facts buried millions of tokens earlier, Gemma 4 31B answered correctly 98 times out of 100, against 82 times out of 100 for the 12B model.

The author has released the test protocol and a single-GPU reproduction built on public software, along with a free license for the package, so the claims can be checked on hardware a smaller lab or team can actually get hold of.