AI Journey
AI status

Event Published ResearchGoogle

A paper has models declare which part of their memory to read, cutting attended tokens by half

A KAIST AI and Google DeepMind paper has models declare which part of their context to read, cutting attended tokens by 52.0% on Gemma-4-31B for a 1.27 point accuracy drop.

What happened

Published here

On 2 September 2026 researchers at KAIST AI and Google DeepMind posted Declarative Attention to arXiv as 2609.02737. The method has the model say, inside its own chain of thought, whether the next stretch of writing needs the whole context, one named region of it, or only what it has written so far; the serving engine reads those declarations the way it reads a tool call and skips most of the stored context instead of scanning all of it. Run zero-shot across 15 long-context tasks on open models taken off the shelf, it cut attended tokens during decoding by 52.0% on Gemma-4-31B and 31.1% on Qwen-3.6-27B, for accuracy drops of 1.27 and 2.75 percentage points, and the paper projects decode wall-clock cost at 0.71 and 0.77 times the usual on a well-optimised serving stack. (arXiv listing and abstract, submitted 2 September 2026)

One decoding step reading the whole stored context, then only a declared region, cutting attended tokens by 52.0%.
The declaration drawn between the panels is what changes how much of the stored context is read to write each token, a quantity nobody bills you for that surfaces instead as latency and as somebody else's serving cost. Diagram: AI Journey.

Open the diagram at full size

Read the full description of this diagram

Two-panel diagram of one decoding step. On the left, every cell of a row standing for the stored context is filled, to show all of it being read to write a single token. Between the panels a box labelled the declaration says the model states inside its own chain of thought whether the next stretch needs the whole context, one named region, or only what it has written so far, and that the serving engine reads that declaration the way it reads a tool call. On the right, the same row has only a middle group of cells filled as the declared region while the rest stays untouched, and the same single token comes out. A table below records attended tokens during decoding down 52.0% on Gemma-4-31B and down 31.1% on Qwen-3.6-27B, for accuracy down 1.27 and 2.75 points.

Tracked on this site: our API price table, where the cache read column lives

What it means for you

Nothing on your invoice changes because of this, and the reason is worth understanding, because it is the same reason long conversations feel expensive. Vendors bill you for context they have seen before as a discounted input token: our table records Claude Fable 5.1 at $10 per million input tokens with cache reads at $0.25, and GPT-6 Astra at $10 with cached input at $1, both read on 5 September 2026. That price covers sending the context again. What this paper cuts is how much of that stored context the model actually reads while it writes each token, which nobody bills separately and which shows up instead as somebody else's serving cost and as your latency. If it holds up, the first thing you will notice is a long chat that answers faster, and the price sheet will be the last thing to move.

In more detail

The numbers, and where each one comes from

  • 52.0% and 31.1% fewer attended tokens during decoding, on Gemma-4-31B and Qwen-3.6-27B, with accuracy falling 1.27 and 2.75 percentage points, from the abstract.
  • 0.71x and 0.77x of the usual decode wall-clock cost on those same models, a roofline projection rather than a measured serving benchmark, from the paper’s HTML version.
  • Up to 98% and 99% of the per-step attention read saved in the focus and local modes at contexts beyond 128K tokens, from the same text.
  • $0.25 against $10, and $1 against $10: what a cache read costs against a fresh input token per million at Claude Fable 5.1 and GPT-6 Astra, as our table recorded them on 5 September 2026.

Why we ran a research paper in a price stream

Because the ledger has a column for exactly the thing being attacked, and the column and the paper do not line up in the way you would expect.

A cache read on a price sheet is what you pay to send a model context it has already been given. It is charged per token sent. Attention over the stored context is what happens next, and it is charged to nobody: the vendor absorbs it as serving cost and you feel it as the wait. So a method that halves the second one is invisible on a bill and obvious in a chat window, and a reader who has been told this makes models cheaper is owed the distinction.

The honest caveats

The wall-clock figure is a projection from a roofline model, not a timing on a live serving stack, and the paper says so. The accuracy cost is small but real, and it was measured zero-shot on open models rather than on the closed ones most readers actually pay for. And a model that declares where it will look can declare wrongly, which is a failure mode with no equivalent in a system that always reads everything.

We are recording it here because it is the kind of change that reaches a price sheet late and quietly, through a cheaper long-context tier rather than through an announcement. Our cache read column is dated. When a cache read price moves, this item is the note it will be dated against.

Sources

Sources read

Written and published by KawsarHow we collect and check data

Cite this

AI Journey, "A paper has models declare which part of their memory to read, cutting attended tokens by half", 6 Sep 2026, https://aijourney.space/updates/2026-09-06-declarative-attention-kv-cache/ (source: arXiv, verified 6 Sep 2026)

Next in the stream

More in Google

All Google news

All logged changes