Small Nepali Legal Language Model NepLEGiT Trained From Scratch on Legal Corpus

Researchers from Kathmandu University published NepLEGiT on 8 August 2026, with the current version indexed on arXiv on 16 September 2026. The paper addresses a plain problem: legal information in Nepal is written in dense, formal Nepali, and legal expertise is scarce outside cities, which leaves many citizens unable to read or interpret the laws that govern them. Rather than adapting an existing large multilingual model, the authors trained a small language model from nothing on legal text alone.

The model is a decoder-based transformer built on the GPT-2 architecture, with 6 layers, 6 attention heads, and a hidden dimension of 384. Training used warmup cosine decay scheduling, gradient accumulation, and mixed precision arithmetic, all standard techniques for fitting a useful model onto limited compute. The training corpus was assembled specifically for the task, drawing on constitutional law, civil and criminal codes, and administrative regulations written in Nepali.

The authors also ran a comparison against two existing multilingual encoder models, mBERT and MuRIL, continuing their pre-training on the same legal corpus rather than training from scratch. mBERT reached a lower validation perplexity than MuRIL on this domain-specific data, which the paper reports as a useful encoder-side baseline that complements NepLEGiT’s own generative design.

The numbers

NepLEGiT was trained on roughly 4 million tokens of curated legal text and has about 30 million parameters, a fraction of the size of the general-purpose models that dominate most public discussion. On the held-out validation split it reached a cross-entropy loss of 0.5684, a perplexity of 1.8, and next-token prediction accuracy of 82.9 percent. Those figures describe how well the model predicts Nepali legal language specifically, not general reasoning or fluency, and they are not directly comparable to benchmarks reported for large multilingual systems.

The paper does not test the model against any deployed product, and no company has adopted it. It is included here as a research signal rather than a product announcement: a demonstration that a narrowly scoped, cheaply trained model can perform competently on a specialized task where large models have little tailored training data. For an answer engine business like Perplexity, whose value depends on routing a query to whatever source answers it best, small specialist models built for a single language and domain are exactly the kind of building block that could sit behind a general interface without needing to be trained by the interface’s own operator.