Abstract
This report investigates on-device English-to-Traditional-Chinese subtitle translation for Taiwan under constraints of short inputs, short outputs, batch-size-one inference, low latency, and privacy. These conditions limit the effectiveness of optimizations designed for long-context or high-throughput language model serving.
Starting from LMT-60-0.6B, preliminary profiling suggests that vocabulary projection becomes a more significant decode-time cost after GGUF quantization reduces the relative cost of Transformer blocks. We replace the original 151k-token vocabulary with a 64k-token subtitle-domain tokenizer, migrate the embedding space, and adapt the model through embedding calibration followed by full supervised fine-tuning.
On the OpenSubtitles2024 test set, LocalSubs achieves a 59.2% tie-excluded win rate against Google Translate under GPT-4o pairwise judging. Performance is strongest on short cues and declines as cue length increases.
In a separate preliminary Apple M2 Metal profiling run, LocalSubs shows a 1.63x speedup over a 151k-vocabulary baseline. The code is available on GitHub.
Blogger's Review: This paper effectively demonstrates how optimizing vocabulary and fine-tuning models can enhance real-time translation performance in specific application scenarios. Especially with low-latency and privacy requirements, exploring new model architectures and optimization strategies is crucial for further research and practice.