Academic Benchmark Finds Google's Gemini 2.5 Flash Leads Bengali Factuality

A paper posted to arXiv on October 13, 2026 set out to answer a question that most English-language benchmarks skip entirely: does a model that writes a good summary in English also write a reliable one in a lower-resource language. The authors, working from Khulna and submitting under Nazifa Tasneem’s name, built a zero-shot test across eight closed-source and open-weight language models, running each one against size-matched English and Bengali test sets drawn from the XL-Sum corpus. Rather than scoring on a single metric, they combined lexical measures such as ROUGE and BLEU with semantic scoring, reference-free factuality checks and a multidimensional LLM-as-a-Judge assessment, a layered approach meant to separate a summary that merely reads well from one that is also faithful to the source article.

The headline finding is a split rather than a single winner. GPT-5.6 Luna topped several English lexical metrics, but that lead did not transfer to Bengali, where LLaMA 3.3 70B produced the strongest ROUGE and BLEU scores instead. On the factuality axis, which the paper treats as the harder and more consequential test, Gemini 2.5 Flash came out ahead for Bengali, posting the highest observed factuality score of the eight models checked. The authors also note that different evaluation methods produced different rankings altogether, which they read as a warning against leaning on any one metric when judging multilingual summarization work, particularly for languages with far fewer labeled resources than English.

The numbers

Eight models were tested across two languages and two size-matched XL-Sum sets, with results split out by lexical score, semantic score and a separate factuality check. GPT-5.6 Luna’s English lexical lead did not hold in Bengali, where LLaMA 3.3 70B took over on ROUGE and BLEU. Gemini 2.5 Flash recorded the highest Bengali factuality score among the group. On Google’s own API price table, Gemini 2.5 Flash is listed at $0.30 per million input tokens and $2.50 per million output tokens, with a 1,048,576 token context window, figures that matter to anyone weighing the factuality result against what the model actually costs to run.

For a team building a Bengali-language news digest, legal summary tool or customer support pipeline, the practical lesson is not to assume an English leaderboard carries over. A model picked purely on English benchmark scores could land a developer with weaker factual grounding in Bengali output, the exact failure mode the paper’s reference-free verification was designed to catch. The paper was accepted for presentation at MCT4SD 2026 in Manila, which should put the full methodology and model list in front of a wider applied audience before the end of the year.