Large multimodal models have made remarkable progress by scaling capacity and training on massive paired multimodal data. Yet this paradigm faces a fundamental bottleneck: aligned multimodal datasets are expensive, hard to cover diverse modality combinations, and limit further advancement. To address this, we introduce Modality Self-Play (MSP), a training framework that composes modalities via proxy text. MSP does not rely on real cross‑modal pairs; instead it uses anchor‑supporting proxy textual descriptions of unseen modalities as semantic bridges. Given an observed modality (e.g., image or audio), the model generates proxy descriptions for the complementary unseen modalities. These descriptions are encoded by contrastively pretrained encoders and aligned with the observed modality through learned bridge layers, enabling cross‑modal interaction without real non‑text supervision. The central hypothesis is that proxy text combined with contrastive encoders provides sufficient semantic structure for multimodal composition to emerge. Experiments show that MSP enables zero‑shot modality composition: models trained only with a real anchor modality and its proxy descriptions can integrate and reason over multiple unseen modalities at inference time. MSP achieves state‑of‑the‑art or competitive results on several multimodal benchmarks. Ablation studies confirm that learned bridges improve alignment and that anchor‑supporting proxy descriptions are essential for effective cross‑modal composition. MSP removes the need for real paired non‑text supervision (e.g., audio‑vision) while still relying on modality‑text pairs, pretrained contrastive encoders, and bridge calibration.
Review