Paper Presents Script-Aware Tokenizer For Indic Languages

Tokenization is the first thing that happens to any piece of text an AI model reads, and it is also the step that decides how much a prompt in a given language actually costs. A new preprint, posted to arXiv on October 8, 2026 and recorded here on October 27, 2026, tackles a problem that has sat quietly under multilingual AI systems for years: tokenizers trained mostly on English corpora tend to break Bengali, Hindi and other Indic-script text into noticeably more pieces than an English sentence carrying the same meaning.

The paper, titled AdaptTok-Indic, extends an approach called adaptive tokenization with features aimed squarely at Indic scripts. It applies Unicode normalization before anything else happens, adds explicit handling for Indic conjunct characters, which are compound letterforms that standard byte-pair tokenizers often split in ways that do not match how a reader would parse the word, and routes text through script-aware logic paired with a learned byte-level boundary predictor. All of this is trained jointly with a small Transformer language model rather than bolted on afterward.

The authors tested the method on Bengali and Hindi as primary targets, with further evaluation on Gujarati, Tamil and Telugu. They compared it against SentencePiece BPE, an existing Indic-oriented tokenizer, and their own implementation of a MAGNET-style baseline.

The numbers

Across five recorded seeds, mean token fertility, a measure of how many tokens a tokenizer needs per word, came to 1.432 for Bengali and 1.388 for Hindi under AdaptTok-Indic, against 2.156 and 2.085 for SentencePiece BPE on the same languages. Recorded downstream task scores averaged 57.69 on the evaluated Indic settings versus 50.82 for the SentencePiece BPE setup. The authors note that independent verification runs using the released prototype reproduced the same directional pattern.

The paper is careful to flag its own limits. It describes the current implementation as a small-scale prototype with documented simplifications, and says no statistical significance testing, no comparison against a released MAGNET model, and no evaluation on code-mixed text, where speakers blend languages within a single sentence, has been carried out yet. Those gaps matter for anyone deciding whether to build on these results now or wait for a larger-scale version with significance testing attached.