OncoNoteBERT Encoder Trained From Scratch On UK Oncology Notes

A team led by Wuraola Oyewusi, with co-authors from the University of Manchester and The Christie, built two oncology-specific language encoders and tested them against two existing clinical models nobody had retrained for this setting. The corpus was a governed set of UK outpatient notes covering lung and head-and-neck cancer care, full of staging language, treatment names, toxicity descriptions and site-specific de-identification markers that a general biomedical model has no reason to have seen.

The headline comparison is between adaptation and rebuilding. RadBERT and PathologyBERT, both trained elsewhere, were tested on the oncology notes with no further training at all, and both fit the text poorly. Continuing to train RadBERT on the oncology corpus, the approach the authors call OncoNote-RadBERT, produced the best raw fit of any model tested. Building a new encoder from scratch, OncoNoteBERT, with its own oncology WordPiece tokenizer, scored worse on that same measure but tokenised clinical terms more efficiently and answered masked-word probes correctly far more often than the continued-pretraining model did.

That split is the paper’s real finding. A model can look weaker by one yardstick and stronger by another, and the authors trace much of the gap to how each tokenizer breaks oncology terms into subword pieces rather than to anything deeper the models learned. Both locally built models also treated an institution’s own de-identification placeholder as a single token, a detail that matters for any health system reusing this approach on its own notes.

The numbers

The corpus spans 290,026 notes from 21,564 patients. Off the shelf, RadBERT posted a perplexity of 113.04 on the oncology text and PathologyBERT posted 2035.03, both signs of a poor fit before any local training. After continued pretraining, RadBERT’s score fell to 2.10, the strongest raw fit in the comparison, while the from-scratch OncoNoteBERT scored 2.83 but got 12 of its 13 masked-token probes right against 7 of 13 for the continued-pretraining model.

The paper is aimed at clinical NLP practitioners rather than general consumers, but it is a useful data point for anyone deciding between a general-purpose language tool and a narrower model built for one institution’s documents, since the two approaches traded wins depending on which number mattered most to the task at hand.