Paper Describes Data-Centric Continual Pretraining Method For Arabic LLMs

The paper addresses a specific cost problem in building Arabic language models: web text scraped for Arabic is noisy, mixes dialects with Modern Standard Arabic (MSA), and continual pretraining on a multilingual base model is expensive when every parameter is updated. The authors propose AQL-Lite, which pairs a curation pipeline, structural cleaning, filtering by heuristics and by a model, MSA selection and deduplication, with a training recipe that touches only part of the network: low-rank adapters placed across layers, the last eight transformer blocks, and the embedding matrix, all under layer-wise learning rates rather than a single global rate.

The central comparison in the paper isolates how much of the benefit comes from the data versus the training method. At a matched 1B-token budget, training on raw, uncurated web text produced no measurable improvement over the untouched base model, while training on the curated AQL data raised a four-benchmark Arabic average by 3.51 points. That result puts the curation step, not the training recipe, as the first lever worth pulling.

A second, 5B-token comparison then tests the training recipe itself against a full-parameter continual pretraining baseline. AQL-Lite reached 94% of the full-parameter run’s benchmark gain while using 52% of its wall-clock time, and beat adapter-only baselines on every benchmark tested. A follow-up ablation found that the adapters, the unfrozen transformer blocks, and the embedding-matrix training each added something on their own, rather than one component carrying the whole effect.

The numbers

• 3.51 points: the four-benchmark Arabic mean gain from curated data over raw web text, at the same 1B-token budget, with raw text alone yielding no gain.
• 94% and 52%: the share of full continual pretraining’s gain AQL-Lite recovered, and the share of its wall-clock time that took, in the 5B-token comparison.
• 75.3: the score reached on the seven-benchmark Open Arabic LLM Leaderboard v2 suite once the recipe was scaled to 40 billion MSA tokens and instruction-tuned.

The paper frames partial-parameter continual pretraining on a carefully curated corpus as a practical route for adapting a multilingual model to a language with sparser resources than English, a framing relevant to any team weighing where to spend a fixed compute budget on language adaptation work.