High‑quality human‑translated parallel corpora improve cross‑lingual transfer, yet their size is usually insufficient for large‑scale pretraining. Machine‑translated web corpora provide parallel data at pretraining scale.
Standard multilingual pretraining (SPT) samples source documents and their translations independently, ignoring the explicit parallel structure. This work investigates parallel pretraining (PPT), which concatenates each document with its translation in the same training sequence.
Models are trained from scratch under matched token budgets using PPT and SPT respectively. Surprisingly, despite the known benefits of parallel data for cross‑lingual tasks, PPT alone does not consistently outperform SPT on general language understanding.
Further analysis shows that PPT builds a stronger cross‑lingual foundation by better exploiting shared information across languages, but the mismatch between parallel training contexts and single‑language downstream inputs limits its gains in generic understanding.
Motivated by this, a two‑stage approach is proposed: first use PPT to establish a cross‑lingual base, then continue with SPT to adapt the model to standard language modeling. In both high‑resource and low‑resource settings, this two‑stage method consistently improves general language understanding and cross‑lingual task performance under equal training budgets.
Review