Nepali Sentiment Benchmark Finds Claude Opus Ahead of Local Encoders

Three researchers, Gaurab Baral, Mimansak Nepal and Grishma Basnet, posted NEPSEE on arXiv on 13 October 2026, a Nepali sentiment benchmark built to fix a problem the paper states plainly: most sentiment datasets carry one annotator per item, so the labels reflect that one person’s habits rather than an agreed judgement. The team collected 5,614 tweets from Nepali public figures and had two annotators label each one blind on a five-point scale, then kept only the 3,189 sentences where both annotators gave the same polarity, discarding roughly four in ten sentences where the two disagreed.

On that three-class, double-agreed set, NepBERTa with a fixed tokenizer scores a macro-F1 of 0.877 and IndicBERTv2 scores 0.867. Three commercial large language models score between those two figures and a touch above them in zero-shot testing, and the paper reports that the gap between the encoders and the commercial models is not statistically significant on this task.

The paper’s second test is harder: a five-point scale where 766 sentences carry two adjacent labels, such as positive and strongly positive, and the authors accept both as correct and train on both rather than forcing a single answer. On that task the paper reports that Claude Opus 5.5 beats the best encoder models by a clear margin in macro-F1, a result the authors attribute partly to the commercial model’s broader pretraining rather than any Nepali-specific tuning.

A separate finding concerns tokenization rather than modelling. The default tokenizers of three Nepali BERT models strip out the non-spacing Devanagari vowel signs and the virama mark before training, a silent preprocessing step. Restoring those marks raises macro-F1 for NepBERTa but lowers it for a model that was pretrained on the stripped text, showing that a tokenizer choice made early in a pipeline can help or hurt depending on what the base model already learned to expect.

The numbers

• 5,614 sentences were labelled by two annotators working blind on the same tweets.
• 3,189 of those sentences survived after sentences with disagreeing polarity were dropped.
• NepBERTa reaches a macro-F1 of 0.877 on the three-class benchmark, against 0.867 for IndicBERTv2.

For a team building Nepali-language products, the practical question the paper raises is whether the cost of running or fine-tuning a dedicated local encoder is worth it once a general model clears the same bar out of the box. Anthropic’s own pricing page lists Claude Opus 5 at $5 per million input tokens and $25 per million output tokens, a cost that a team can now weigh directly against the accuracy difference this benchmark reports, rather than guessing at it from English-language results.