New Preprint Proposes Compact Recurrent Cell Named baseVultra

Recurrent neural networks built from gated cells like the LSTM and the GRU have mostly been pushed aside by transformers for large-scale language work, but they remain a practical choice for streaming sequence tasks where processing has to happen one step at a time with a fixed, small memory footprint. The new preprint asks a narrow design question inside that smaller world: instead of spending most of a cell’s parameters on the gates that control what gets remembered, what happens if most of that budget goes to the branch that generates the candidate update itself.

The proposed cell, called baseVultra, keeps one dense H by H matrix for candidate generation and swaps the usual dense gate matrices for lighter diagonal interactions, adding two small auxiliary branches, a running average cache, and a learned way of blending outputs. The gates also get limited access to the state from two steps back through simple diagonal terms, a cheap way to give them a bit of longer memory without adding a full matrix.

On TinyShakespeare, a small character-level text dataset, baseVultra edges out parameter-matched LSTM and GRU cells on validation loss while using noticeably less dense computation than GRU at matched hidden size. On text8, a dataset the paper describes as roughly forty five times larger, the advantage narrows to roughly a tie with the older cells. On synthetic formal-language tests designed to probe how well a cell tracks structure with little memory to work with, baseVultra’s advantage is largest when the hidden state is small and fades as the hidden size grows, which the authors read as the central finding: this reallocation of capacity helps most exactly when memory is scarce, and stops mattering once a model has room to spare.

The numbers

The paper reports a validation loss of 1.594 for baseVultra against 1.604 for a matched LSTM and 1.616 for a matched GRU on the TinyShakespeare task, with baseVultra’s dense recurrent computation running at about 54% of GRU’s under the same hidden size. On the roughly 45x larger text8 benchmark the gap mostly disappears, with baseVultra at 1.854 bits per character against 1.849 for GRU and 1.837 for LSTM. The paper notes it was revised after review feedback from an academic conference, with one weak accept and two borderline scores recorded in its submission history.