In pre-trained large language models (LLMs), a fixed tokenizer allocates vocabulary based on the pre-training corpus, reflecting the deployment priorities at that time. As those priorities shift, languages added later are often split into many more tokens, which increases latency, compute, and energy consumption for users of those languages. Cloud models can afford a broad vocabulary since the embedding and LM-head matrices constitute a small fraction of their parameters. In contrast, compact models have these matrices as a significant share of per-token decode bandwidth, leading on-device models to ship small vocabularies and accept fragmentation outside a fixed language set.
We present a tokenizer expansion method for upgrading a pre-trained model's tokenizer when the model producer controls its design. This method continues existing BPE merges on a multilingual corpus, allowing most source tokens to carry over unchanged as single tokens, while each new token has an exact decomposition into source tokens. We copy the carried-over embedding rows unchanged and initialize new rows as the mean of their source sub-token embeddings. A two-stage adaptation, embedding-only training followed by full-model continued pre-training, recovers source-checkpoint quality. We apply this recipe to a continued pre-trained checkpoint of LFM2-8B-A1B, an 8B-parameter Mixture-of-Experts model, to help produce LFM2.5-8B-A1B with a 128K tokenizer. The expanded tokenizer encodes Hindi and Vietnamese in roughly $2.4\times$ and $2.6\times$ fewer tokens than the source (up to $4.0\times$ on Thai). Combining these reductions with the measured per-token cost of the larger vocabulary, we estimate a $2.2$-$3.7\times$ per-character decode speedup for these languages across our reference devices. We release the model weights and the expanded tokenizer, and report the negative findings that shaped the recipe.
Blogger's Review: This research presents a significant optimization strategy for the application of large language models. By improving tokenizer design, it not only enhances the efficiency of multilingual support but also effectively reduces resource consumption. The potential for real-world applications is worth further exploration.