Study Finds Tiny Chinese BERT Models Favor MLM Over MacBERT

A paper posted to arXiv on October 6, 2026 reports a controlled comparison of three pretraining strategies for Chinese BERT models: standard Masked Language Modeling (MLM), Whole Word Masking (WWM), and the synonym-replacement approach used by MacBERT. The author, Yiping Bai, trained three models from an identical four-layer, 256-hidden-dimension architecture with 8.7 million parameters, using the same 1.29 million-sentence corpus drawn from Chinese Wikipedia and identical hyperparameters, so that only the masking strategy differed between runs.

Most prior work testing these three strategies against each other used base-scale models of 110 million parameters or more, where the published ranking favors MacBERT over WWM over MLM. At the much smaller scale tested here, that ranking reversed. MLM won three of the five intrinsic evaluation dimensions the paper tracks, which are perplexity, MLM hit rate, semantic discrimination, grammatical judgment and contextual sensitivity. WWM came out ahead on perplexity and hit rate specifically, while MacBERT, hampered by a synonym dictionary covering only 222 entries and 3.3 percent of tokens, produced a model whose perplexity was far worse than either rival despite recording the lowest training loss of the three during training.

That last detail is the paper’s sharper warning. A model’s own training loss is often used as a stand-in for how good it will be once deployed, and here the model with the best loss number produced the worst perplexity score once evaluated, a pitfall the author ties specifically to mixed replacement strategies like MacBERT’s.

The numbers

• WWM perplexity: 1.27, well below MLM’s score.
• MLM perplexity: 2.10, the figure WWM improved on.
• Perplexity improvement: 39.5%, WWM’s gain over the MLM baseline.
• MacBERT perplexity: 47.23, far above either rival despite its lower training loss.

For a reader choosing how to pretrain a small model meant to run on a phone or at the edge rather than in a data centre, the lesson is specific rather than general: a masking recipe copied from a paper that tested a model with 110 million or more parameters may be the wrong choice once the parameter budget drops under nine million, and checking training loss alone before shipping a tiny model would have hidden exactly the failure this comparison caught. The author has released the trained models and corpus publicly, which lets other teams rerun the same five tests on their own tiny architectures rather than trusting a single paper’s ranking as a given across every model size.