Paper Traces How Mistral and Peer Models Encode Three-Token Relationships

Researchers led by Gurbir Arora, with coauthors including Christopher J. Earls, posted a paper on arXiv on September 28, 2026, that looks inside transformer models to see how they build up an understanding of relationships between tokens rather than just responding to tokens one at a time. The team fed models prompts built around cyclic concepts, months, hours, weekdays and musical notes, that require inferring how three tokens relate to each other before the model can correctly predict what comes next. They then examined the internal activations layer by layer across model families: Llama, Qwen, Gemma and Mistral.

The finding that runs across every family, Mistral included, is a layerwise progression in how the model organizes and uses that relational information. In the middle layers, the model builds a joint representation based on the relationship between just two of the three tokens. Only in later layers does it assemble a representation that ties all three tokens together, and it is that fuller representation that the model actually relies on to complete the task correctly. The paper also reports other token relationships that are geometrically structured inside the model’s activations but play no causal role in the next-token prediction, meaning the model appears to represent more structure than it actually uses.

One further result stands out for anyone building on this kind of interpretability work: when the researchers restricted the models to using only the causally relevant joint representations they had identified, next-token prediction accuracy improved rather than degraded. That suggests some of what a model computes internally is either redundant or actively unhelpful for the task at hand, at least for this class of compositional cyclic reasoning.

The numbers

The paper studied Llama, Qwen, Gemma and Mistral, testing cyclic concepts spanning months, hours, weekdays and musical notes. The paper itself runs 46 pages and carries 28 figures documenting the geometric and causal analysis across those families and concepts.

Work like this is typically done through the model’s own weights or a hosted API, and for Mistral that means a per-token bill. Mistral Large 3’s international endpoint rate is $0.50 per million input tokens and $1.50 per million output tokens, against $1.50 and $7.50 for Mistral Medium 3.5, a gap that matters for a research group running the thousands of short probing prompts this kind of activation study needs rather than a handful of long generations.