The rapid progress of large foundation models has predominantly relied on pretraining with large-scale text corpora. However, various forms of knowledge are conveyed through visual representations, where figures, typeset equations, and page layouts carry rich information that cannot be fully captured by text alone.
Current pretraining approaches discard these visual cues by converting visually rich sources, such as documents and web pages, into plain text for learning language intelligence. This paper challenges the assumption that language models must be trained solely on text representations and demonstrates that Visual Pretraining is a scalable learner for foundation model intelligence.
To this end, we conduct a systematic study of unsupervised visual pretraining paradigms that leverage visual documents without text extraction. Across multiple backbones and benchmarks, visual pretraining on the same underlying corpora consistently outperforms text-only pretraining, offering an efficient pathway to scalable language intelligence.
Blogger's Review: This research provides a fresh perspective on training language models, highlighting the significance of visual information. In the future of AI development, multimodal learning that combines visual and textual data will be a crucial direction for enhancing model intelligence. By utilizing visual pretraining, models can gain a more comprehensive understanding of information, driving advancements in language intelligence.