Paper Proposes Reversible Tokenizer That Cuts Vocabulary Bloat

The Functionalizer, described in an arXiv paper submitted July 7, 2026, tackles a problem every subword tokenizer inherits: hello, Hello, HELLO and Héllo end up as separate vocabulary entries, or the differences between them get normalized away and lost entirely. Authors Connor Makowski and Willem Guter propose splitting a word into a canonical base token plus a stream of reversible transformation opcodes, stored in the Unicode Private Use Area, that record casing, diacritics and character repetition separately from the base spelling.

The scheme adds a CAPITALIZE opcode, thirteen dedicated diacritic opcodes, and repeated-character operators called REPEAT and MULTIREPEAT, all fully reversible so the original text can be reconstructed exactly. Because the transformation is factored out rather than baked into separate vocabulary slots, the tokenizer needs fewer distinct tokens to cover the same text.

The numbers

Tested across six natural language and code corpora, the Functionalizer achieved complete coverage while cutting actual vocabulary slot requirements by up to 16 percent under unconstrained conditions. The paper also reports a domain split: the scheme compresses indentation-heavy code sequences but produces longer sequences for natural-language prose, because prose carries less of the repeated structure the opcodes capture. In preliminary downstream tests on a 25M-parameter GPT-2-scale model, the approach improved code syntax validity and reduced code character perplexity, while text coherence on prose stayed roughly comparable to a standard tokenizer.

Why this matters beyond the lab bench comes down to cost and context budget. Every token a model processes costs compute and eats into the context window, so a smaller vocabulary that still covers a corpus completely is a direct lever on both. Perplexity AI runs an answer engine that indexes and reasons over live web text, exactly the kind of mixed-case, code-and-prose corpus this paper tests against, which makes tokenizer efficiency a practical operating question rather than an academic one, even though the paper’s authors are not affiliated with Perplexity. The tradeoff the researchers flag, better for code and worse for prose sequence length, is the kind of detail a team would need to weigh before swapping a production tokenizer, since prose inflation could offset the vocabulary savings depending on how much of a system’s traffic is prose rather than code.

The authors describe the results as preliminary, run at a research scale well below production language models, and call for further validation at production scale before drawing firm conclusions about how the approach holds up once vocabulary size, corpus diversity and model scale all grow together.