Real‑world outpatient oncology notes contain specialised terminology, tumour‑staging expressions, treatment names, toxicity descriptions, and institution‑specific de‑identification markers that are often poorly represented by generic biomedical or clinical language models. We built and evaluated oncology‑specific BERT‑style encoders on a governed UK outpatient oncology corpus (290,026 notes from 21,564 lung and head‑and‑neck cancer patients). The study compared two public encoders—RadBERT and PathologyBERT—with two locally‑developed strategies:
- OncoNote‑RadBERT: continued masked language model (MLM) pre‑training on RadBERT;
- OncoNoteBERT: trained from scratch using an oncology‑domain WordPiece tokenizer.
Evaluation metrics comprised MLM loss and perplexity on a validation set, tokenizer fragmentation statistics, clinical‑term tokenisation quality, masked‑token probes, and exploratory representation analysis. In zero‑shot evaluation, external encoders fit the oncology corpus poorly (RadBERT perplexity = 113.04; PathologyBERT = 2035.03), whereas continued pre‑training yielded the lowest perplexity (2.10 for OncoNote‑RadBERT). OncoNoteBERT achieved a perplexity of 2.83 while delivering the most efficient tokenisation—lower subword fertility and shorter normalised sequence length. For the masked‑token probes, OncoNoteBERT correctly predicted 12 of 13 tokens, compared with 7 of 13 for OncoNote‑RadBERT. This gap was partly attributable to tokenizer fragmentation rather than learned semantics alone. Both locally‑developed models encoded the institutional placeholder as a single learnable token. The findings demonstrate that continued adaptation and bespoke tokenisation provide complementary benefits, and that representation‑layer design matters before applying adjacent‑domain encoders to oncology NLP.
Review