Paper Adds Input-Dependent Readout Phase To Resonant Sequence Models

A new paper posted to arXiv on September 26, 2026 addresses a design choice that sits under every sequence model built to avoid the quadratic cost of attention. Such models keep a fixed-size state rather than growing one with the sequence, and the state can be updated either with an input-dependent transition, which gives content-dependent memory but forces slower FFT-based training, or with a fixed transition, which keeps a fast convolutional training path but reads memory the same way regardless of what the model has seen. ResonatorLM, a recent model of the second kind, reads a bank of damped complex oscillators through a phase that is learned once per head and then fixed.

The paper, by Shivesh Rane, modifies only the read step. The phase that reads the oscillator bank becomes a function of the current token rather than a fixed per-head value, which adds input dependence without touching the state transition itself, so the exact convolutional training path survives unchanged. The authors report the change costs 0.2 percent more parameters.

An analysis of the untrained oscillator bank found that its phase does encode elapsed time, but an ordinary token’s signal falls below background noise within 2 to 8 steps, and no choice of damping extended linear reconstruction of position past roughly 16 tokens, a ceiling the fixed phase cannot escape on its own.

The numbers

Tested on 6M-parameter character-level models trained on WikiText-2 with a training length of 256 tokens, the original fixed phase matched the new input-dependent phase on sequences at that training length but broke down once evaluated on 1024-token sequences, where perplexity reached 12.81. The input-dependent phase, evaluated at the same 1024-token length, brought perplexity down to 3.76. The authors trace the original failure to long-memory heads that accumulate more history at test time than they were exposed to during training, and note that the learned phase tracks the current character rather than distance, which is why it does not improve retrieval even as it fixes the length generalization problem. The result separates two properties that are often bundled together in fixed-state sequence models: how well a model generalizes to a longer sequence than it trained on, and how well it can retrieve something from far back in that sequence.