Deep neural networks achieve impressive results on vision benchmarks, yet they remain vulnerable to distribution shift, spurious correlations, texture bias, adversarial perturbations, and forgetting in continual learning. Conventional supervision relies on one‑hot labels, which provide little inter‑class semantic structure. We investigate pretrained language models as lightweight auxiliary supervision that requires no image‑text pairs, prompt engineering, or contrastive objectives.
Two forms of language guidance are introduced: Explicit Language Guidance (ExLG) and Implicit Language Guidance (ImLG). A comprehensive evaluation spans six generalization regimes— in‑distribution, out‑of‑distribution, shortcut and spurious correlation resistance, texture versus shape bias, adversarial robustness, and continual learning. Findings reveal complementary strengths: ExLG consistently improves in‑distribution performance, low‑data scenarios, and retention in continual learning, while ImLG excels in shortcut‑sensitive settings and under strong adversarial attacks. Both approaches add minimal parameters and training overhead. The analysis clarifies when language‑derived structure benefits supervised vision training and offers a practical roadmap for leveraging off‑the‑shelf pretrained models from another modality as auxiliary supervision.
Review: This study highlights the promise of cross‑modal language signals for enhancing the generalization of vision models, providing a lightweight alternative to large‑scale paired data and opening new avenues for multimodal supervision.